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SUMMARY 


Objectives 

The  objectives  of  this  evaluation  study  were:  (1)  to  evaluate  the  training  effectiveness  of  the 
Automated  Flight  Training  System  (AFTS)  in  F4  Combat  Crew  Training.  (2)  to  identify  desired  hardware 
and  software  modifications  for  operational  devices  and  (3)  to  identify  effective  methods  of  operational 
training  utilization  of  the  AFTS.  While  the  capabilities  of  the  AFTS  include  adaptive  training  features,  the 
present  research  was  not  an  evaluation  of  adaptive  training  per  se,  but  rather  an  evaluation  of  an  entire 
system  in  which  adaptive  training  represented  only  one  of  its  many  features. 

Approach 

Subjects  were  24  students  from  Class  75-CRI.  at  Luke  AFB.  Arizona,  divided  into  two  groups  of  12 
each.  All  were  recent  graduates  of  undergraduate  pilot  training  who  had  been  assigned  to  F-4  combat  crew 
training  in  Operational  Training  Course  F4000H. 

The  AFTS  is  a parasitic  device  attached  to  an  existing  I 4F.  simulator.  The  host  simulator  was  used  in 
normal  F4  training  and  did  not  have  a visual  capability.  The  AFTS  is  designed  to  automatically  direct 
pilots  through  a series  of  ground  controlled  approach  (C.CA)  and  tactical  air  navigation  (TACAN)  exercises. 
U consists  of  a computer  system,  a voice  genciation  system,  a display  and  control  system,  and  a system  for 
interfacing  with  the  host  F4F  simulator. 

One  group  of  subjects  received  GCA  training  using  the  AFTS.  while  the  other  group  received 
conventional  GCA  training  administered  by  instructor  pilots  and  simulator  operators.  Students  were 
randomly  paired  so  that  one  member  of  a pair  was  assigned  to  the  AFTS  Group  and  the  other  member  to 
the  standard  GCA  training  group.  The  AFTS  group  pair-member  received  his  GCA  training  before  his 
Standard  GCA-trained  counterpart.  The  pair-member  assigned  to  the  Standard  Group  received  the  same 
presentation  of  difficulty  levels  within  the  adaptive  training  schedule  as  his  AFTS-t  rained  counterpart  to 
avoid  the  potential  criticism  of  unequal  training  provided  both  groups. 

The  GCA  training  was  given  during  the  First  1?  sorties  in  the  F4  simulator  training  syllabus. 
Independent  variables  were  a set  of  objective  performance  measures  generated  automatically  by  the  AFTS 
for  all  subjects  and  were  collected  on  all  training  sorties  and  on  two  sets  of  criterion  sorties.  The  criterion 
sorties  consisted  of  three  GCAs  each,  the  First  set  administered  immediately  after  initial  training  and  the 
second  set  administered  approximately  four  months  after  the  lirst  criterion  sorties.  In  addition  to  the 
objective  performance  measures,  student  and  instructor  questionnaire  data  and  maintenance  data  were 
collected  and  analyzed. 

Results 

The  results  ol  the  evaluation  are  presented  and  discussed  according  to  three  topical  areas:  (I)  the 
training  el  fee  tiveness  of  the  system.  (2)  qualitative  evaluation  of  the  system,  and  (3)  operational  reliability 
and  maintainability  of  the  system. 

Analyses  of  the  training  effectiveness  results  of  the  evaluation  study  showed:  (1)  there  were  no 
differences  in  performance  on  the  training  trials  between  the  AITS  and  Standard  Groups.  (2)  the  group 
trained  by  a machine  controller  performed  as  well  as  the  group  trained  by  human  operators  on  the  basis  of 
the  criterion  sorties,  and  (3)  the  AI  TS  d<x*s  not  appear  to  train  any  adverse  GCA  responses  as  measured 
during  the  study. 

The  results  for  the  qualitative  evaluation  of  the  system  indicate  that  students  and  instructors  liked  the 
AI  TS  as  a training  device.  Suggested  changes  included  speeding  up  the  transmission  rate  of  information 
given  during  the  GCA  and  improving  message  priorities  to  be  more  like  real  GCA  controllers.  The  AFTS 
features  most  liked  were  the  replay  capability  and  the  performance  measurement  hardcopy  printout. 

Reliability  and  maintainability  data  collected  during  the  evaluation  indicated  that  (l)the  monitor 
buffer  interlace  device  should  be  improved  through  use  of  better  manufacturing  materials  or  redesign;  (2) 
the  replay  system  of  the  AFTS  was  found  to  have  software  problems  which  required  modification;  and  (3) 
the  4010  Tektronix  displays  which  were  used  for  operator  control  and  replay  were  wearing  out  rapidly 
under  heavy  use,  and  it  was  suggested  that  other  display  systems  be  examined  to  minimize  this  problem. 


tv? 


CU)V*.,W*W*»i*>«*»  .-r 


1 


i 


Implications 

The  results  suggested  that  tiiere  were  more  dilTiculty  levels  provided  by  the  AFTS  than  could  be  used 
for  effective  training  in  the  F-4  program.  The  mechanization  of  the  adaptive  scheduling  algorithm  possesses 
several  characteristics  which  appear  to  require  further  empirical  validation  prior  to  operational 
implementation.  Questions  were  raised  regarding  optimizing  the  use  of  the  AFTS  in  maintaining  Hying  skill 
once  a student  has  completed  the  adaptive  program.  It  was  noted  that  the  question  of  whether  additional 
Automated  Flight  Turning  Systems  should  be  procured  was  not  addressed  in  the  present  study  and  that 
system  costs  should  be  analyzed  in  terms  of  potential  system  application  to  answer  the  procurement 
question. 
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USAF  EVALUATION  OF  AN  AUTOMATED  ADAPTIVE  FLIGHT  TRAINING  SYSTEM 


I.  INTRODUCTION 


In  August  1()73,  the  Tactical  Air  Command  (TAC)  began  acceptance  of  an  Automated  Flight  Training 
System  (AFTS)  built  by  Logicon,  Inc.  The  device,  installed  as  a parasitic  system  on  one  of  the  existing  F-4E 
simulators  at  Luke  AFB,  Arizona  was  designed  to  provide  automated  adaptive  training  for 
ground -con  trolled  approaches.  In  December  1973,  TAC  requested  that  AFHRL  conduct  an  operational 
evaluation  of  the  AFTS  in  the  F4  combat  crew  training  program.  Through  mutual  agreement  of  both  TAC 
and  AFHRL,  the  evaluation  was  initiated  in  May  1974.  The  major  objectives  of  the  evaluation  were:  (1)  to 
evaluate  the  training  effectiveness  of  the  Automated  Flight  Training  System  (AFTS)  in  the  F-4  Training 
Program,  (2)  to  identify  desired  hardware  and  software  modifications  for  operational  devices,  and  (3)  to 
identify  effective  methods  of  operational  training  use.  Since  one  of  the  major  characteristics  of  the  AFTS 
was  its  use  of  adaptive  training,  a brief  description  of  the  concept  and  related  research  literature  will  be 
presented. 


Adaptive  Training  Defined 


The  term  “Adaptive  Training,”  typically  is  used  to  represent  a training  situation  . . . “in  which  the 
problem,  the  stimulus,  or  the  task  is  (automatically)  varied  as  a function  of  how  well  the  trainee  performs,” 
(Kelley,  1971).  It  can  be  seen  from  this  definition  that  adaptive  training  required:  (1)  “A  continuous  or 
repetitive  measurement  of  trainee  performance,"  (2)  “One  or  more  task  variables  that  can  be  adjusted  to 
change  task  difficulty,"  and  (3)  “A  means  for  automatically  adapting  task  difficulty  as  a function  of  the 
performance  measurement  such  that  the  task  becomes  more  difficul;  as  the  trainee  becomes  more  skilled,” 
(Kelley  and  Wargo,  1968). 


In  most  instances  the  use  of  the  term  “Adaptive  Training”  refers  to  a training  situation  in  which  a 
trainee  works  with  a device  to  help  him  acquire  a skill.  The  properties  of  the  device  are  such  that  the  trainee 
receives  a series  of  practice  exercises,  the  difficulty  of  which  is  automatically  adjusted  according  to  how 
well  the  trainee  performs.  This  trainee-device  interaction  is  similar  to  a non-mechanized  learning  situation  in 
which  . . . “The  skilled  instructor  varies  the  difficulty  of  the  tasks  he  gives  to  a student  as  a function  of  how 
well  that  student  has  been  performing  ...”  (Kelley,  1969). 


While  the  development  of  an  adaptive  training  device  which  can  duplicate  the  performance  of  a 
skilled  instructor  is  a dramatic  achievement  by  itself,  the  potential  value  of  adaptive  training  as  an 
educational  strategy  seems  far  greater.  Properly  designed,  adaptive  training  provides  individualized 
instruction  that  is  characteristically  more  accurate  and  reliable  than  that  which  most  skilled  instructors  can 
provide.  It  can  be  adjusted  and  continuously  refined  to  generate  the  skill  required,  and  because  of  its 
mechanization  requirements,  can  be  computer  managed  once  an  optimized  program  is  determined,  thus 
obviating  the  need  for  further  instructional  involvement  by  a skilled  instructor. 


Additional  Considerations 


It  has  been  noted  that  adaptive  training  is  based  on  the  assumed  training  effectiveness  of 
automatically  increasing  training  task  difficulty  as  the  trainee’s  performance  improves,  so  as  to  match  his 
increasing  performance  capabilities.  This  concept  is  related  to  an  older  notion  of  reading  readiness  in  which 
the  young  child  is  prepared  to  be  ready  to  learn  to  read  by  the  time  he  starts  the  first  grade.  In  practice,  of 
course,  some  children  are  more  ready  than  others.  It  is  suggested  that  the  adaptive  trainee,  just  like  the  first 
grader  learning  to  read,  will  not  learn  a new  skill  until  he  is  ready.  Most  trainees  begin  practice  with 
different  individual  skill  levels,  or  readiness,  and  as  a result  may  not  benefit  maximally  from  the  same 
sequence  of  training  tasks,  regardless  of  the  properties  of  the  schedule  which  determines  how  task  difficulty 
increases.  It  should  be  pointed  out,  however,  that  this  potential  problem  is  inherent  in  all  learning  and 
training  situations. 


Another  potential  problem  exists  in  some  characterizations  of  task  difficulty  where  the  training  task 
is  dimensionalized  on  the  basis  of  face  validity  or  convenience  of  mechanization.  In  aviation  training,  this 
takes  the  form  of  an  initially  simplified  training  task  to  which  environmental  variables,  such  as  air 
turbulence  or  wind  direction,  arc  manipulated  to  generate  levels  of  apparent  increasing  difficulty.  This 
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practice  is  not  unreasonable  and  seems  to  be  based  on  the  assumption  that  representativeness  is  equivalent 
to  difficulty.  For  example,  the  real  world  lias  cross  winds  ami  rough  air;  therefore,  these  conditions  create 
more  difficulty  for  a pilot  landing  an  aircraft  than  wind  down  the  runway  and  smooth  air.  Such  notions 
seem  reasonable  to  the  skilled  instructor.  Whether  this  scaling  of  task  difficulty  is  appropriate  for 
manipulating  the  practice  of  naive  trainees  is  an  empirical  question. 


Another  potential  problem  area  concerns  the  role  of  the  instructor.  Traditionally,  he  performs  a 
number  of  teaching  functions:  testing  and  scoring  student  performances,  pointing  out  errors,  selecting  the 
next  training  tasks  or  assigning  remedial  practice.  Adaptive  training  programs,  however,  accomplish  these 
tasks  without  an  instructor  pilot.  In  a sense,  the  use  of  adaptive  training  may  obviate  many  of  the 
instructor’s  favorite  roles.  What  the  instructor  still  will  be  needed  to  do  is  relate  the  trainee's  learning  to  the 
circumstances  in  the  real  world  in  which  the  newly  acquired  skills  will  be  needed  and  how  they  may  best  be 
employed.  The  relating  of  instruction  to  the  real  world  is  a phase  of  teaching  that  is  frequently  overlooked 
because  of  the  obvious  focus  on  the  trainee’s  need  to  acquire  the  skill  in  the  first  place.  Thus,  the  instructor 
may  find  many  of  his  old  familiar  and  favorite  teaciting  tasks  no  longer  needed  and  in  addition  may  find  he 
has  to  learn  to  perform  new  and  unfamiliar  functions  in  the  total  training  system.  It  may,  therefore,  be 
reasonable  to  suspect  that  the  instructor  may  experience  certain  frustrations  in  his  involvement  with 
adaptive  training. 


Adaptive  Training  in  Aviation 

Adaptive  training  concepts  have  been  extended  to  aviation  and  have  resulted  in  the  development  of 
several  automated  systems.  To  date,  studies  of  adaptive  training  in  aviation  have  focused  on  demonstrations 
of  feasibility. 


Charles  and  Johnson  (1971)  developed  an  automated  ground  controlled  approach  (GCA)  training 
program  for  the  Navy.  This  computerized  system  was  the  forerunner  on  the  F-4  Automated  Flight  Training 
System  (AFTS)  at  Luke  AFB.  Arizona,  which  was  evaluated  in  the  present  effort.  The  program  was 
developed  for  the  Training  Device  Computer  (TRADEC)  System  at  the  Naval  Training  Fquipment  Center. 
Orlando.  Florida.  A GCA  Right  segment  was  selected  as  the  initial  training  task.  Procedures  for  automated 
data  collection  recording,  and  student  record  keeping  were  programmed  into  the  system.  All  operations  of 
the  system  were  performed  automatically,  including  on-line  structuring  of  tlv.  training  course  as  a function 
of  student  performance.  Twelve  operational  F4  pilots  were  utilized  for  (Ik  demonstration.  Pilot  opinion 
indicated  that  the  system  did  rcRcct  operational  GCA  requirements  and  would  be  beneficial  in  operational 
training  systems.  It  should  again  be  emphasized  that  the  study  was  primarily  a feasibility  demonstration  of 
the  capability  to  automate  GCA  training  and  not  a comparative  evaluation. 


Charles  et  al.,  (1973)  later  applied  the  adaptive  training  techniques  to  the  acquisition  of  basic 
instrument  Right  skills.  Again,  the  study  was  performed  using  the  TRADEC  system  at  the  Naval  Training 
Equipment  Center.  Basic  instrument  Right  maneuvers  for  the  F4  aircraft,  straight  and  level,  climbs  and 
descents,  level  turns,  and  climbing/desccnding  turns  were  automated.  variables  such  as  maneuver 
difficulty,  aircraft  weight,  center  of  gravity,  and  atmospheric  turbulence  were  used  to  control  task 
difficulty.  Four  trainees  representing  a wide  variety  of  aviation  skills  were  given  training  using  the 
automated  instrument  Right  training  program.  None  of  the  trainees  were  operationally  qualified  F4  pl’ots. 
Training  was  conducted  in  one  hour  sessions  with  each  student  completing  as  many  runs  as  possible.  Each 
successive  flight  began  where  the  preceding  one  had  terminated.  Progress  and  updating  were  automatically 
maintained  by  the  computer  program.  The  authors  concluded  that  an  automated  syllabus  for  training 
instrument  Right  maneuvers  could  be  implemented  and  that  a student  performance  score  refiecting 
operational  standards  could  be  developed. 

Charles  et  al.,  (1972)  also  conducted  a feasibility  demonstration  on  the  application  of 
automated-adaptive  training  techniques  for  air-to-air  intercept  training  in  the  TRADEC  Right  simulator 
configured  as  an  F4.  The  training  task  included  three  phases;  (1)  a climb  task  under  GC1/C1C  control,  (2) 
an  attack  phase  under  RIO  control,  and  (3)  a descent  phase  also  under  GCI/CIC  control.  Missile  intercepts 
including  head-on,  forward-quarter,  and  beam  runs  were  incorporated  into  a training  syllabus.  Atmospheric 
turbulence,  aircraft  configuration,  and  bank  angle  were  employed  as  adaptive  variables.  Performance  was 
measured  objectively  for  each  phase,  and  the  syllabus  was  restructured  based  on  student  performance.  Since 
the  study  was  designed  primarily  to  demonstrate  the  feasibility  of  automated  air-to-air  training,  only  three 
subjects  were  used.  The  results  established  the  technical  feasibility  of  the  training. 
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As  tins  brief  review  indicates,  published  reports  to  date  have  only  documented  the  technical 
feasibility  of  adaptive  training  programs.  No  studies  have  been  completed  which  compare  these  adaptive 
training  programs  with  conventional  training.  The  Navy  had  planned  an  evaluation  of  its  version  of  the 
Automated  Flight  Training  System,  but  at  the  time  of  this  writing,  the  results  were  not  available  (Puig  ct  aL 
1974).  Consequently,  the  present  evaluation  is  one  of  the  first  studies  to  compare  an  operational  adaptive 
training  program  with  conventional  training  techniques.  It  should  be  emphasized  that  the  present  research 
was  not  an  evaluation  of  adaptive  training  per  se,  but  rather  an  evaluation  of  an  entire  system  in  which 
adaptive  training  represented  only  one  of  its  many  features. 


II.  METHOD 


Subjects 

Twenty-four  students  from  Class  75-CRL  at  Luke  AFB,  Arizona,  participated  in  the  study.  All 
students  were  recent  undergraduate  pilot  training  (UPT)  graduates  who  had  been  assigned  to  F4  combat 
crew  training  (CCT)  in  Operational  Training  Course  F4000B.  All  students  recently  had  completed  the  TAC 
Tactical  Fighter  l.cad-ln  course  prior  to  arrival.  Four  students  had  previous  experience  in  the  F4  as 
weapons  systems  officers  (WSO). 

AFTS  System  Design 

The  Automated  Flight  Training  System  (AFTS)  was  designed  to  automatically  direct  pilots  through  a 
series  of  GCA  and  TACAN  approach  training  exercises.  The  system  will  be  briefly  discussed  according  to  its 
hardware  and  software  components.  A complete  description  may  be  found  in  the  System  Operation  Manual 
for  the  AFTS  (Logicon,  1974). 

Hardware.  The  AFTS  was  modularly  designed  to  operate  as  a parasitic  device  attached  to  an  existing 
F4K  flight  simulator.  The  host  simulator  was  used  in  the  normal  F4  training  program  and  did  not  have  a 
visual  capability.  The  AFTS  configuration  consisted  of:  (1)  i computer  system  including  a Data  General 
Nova  800  digital  computer,  card  reader,  line  printer,  magnetic  tape  drive,  and  two  memory  discs;  (2)  a 
Mctioiab  voice  generation  system;  (3)  two  Tektronix  4010  graphic  CRT  consoles,  one  used  for  AFTS 
control  and  the  other  for  student  GCA  replay,  and  (4)  the  monitor  buffer  and  Y -switch,  the  hardware 
interfaces  of  the  AFTS  with  the  host  F-4H  simulator. 

Software.  The  AFTS  provided  three  modes  of  training.  However,  only  Mode  1 was  used  for  the 
evaluation.  Tins  mode  provided  automatic  adaptively  scheduled  GCA  training.  The  GCA  syllabus  was  stored 
in  a table  format  and  organized  in  increasing  levels  of  difficulty.  Seventy-six  different  difficulty  levels  were 
provided.  Difficulty  levels  were  varied  as  a function  of  wind  direction  and  speed,  aircraft  weight, 
atmospheric  turbulence,  and  aircraft  emergencies.  Difficulty  level  incremented  or  decremented  based  upon 
pilot  performance  on  a set  GCA  task.  A scoring  algorithm  provided  the  basis  for  changes  in  difficulty  levels. 

Mode  2 provided  a representative  set  of  ten  difficulty  levels  of  AFTS  GCA  exercises.  The  steps  were 
ananged  in  increasing  levels  of  difficulty,  and  provided  instructor  personnel  the  capability  to  familiarize 
themselves  with  the  performance  requirements  of  AFTS  GCA  runs.  During  familiarization  approaches, 
ground  controller  voice  commands  were  given  to  a simulated  pilot.  At  the  completion  of  the  run,  a 
hardcopy  listing  of  the  simulated  pilot  performance  was  furnished  from  the  line  printer.  The  adaptive 
scheduling  algorithm  was  bypassed  in  this  mode. 

Mode  3 provided  one  difficulty  level  of  an  initial  TACAN  approach.  A final  approach  consisting  of 
cither  a TACAN,  ASR,  or  PAR  could  be  selected.  A complete  description  of  the  performance  measurement 
and  adaptive  scheduling  algorithms  is  presented  in  Appendix  A. 

Performance  Measurement 

Both  subjective  and  objective  performance  measures  were  collected.  The  subjective  measurement 
consisted  of  a questionnaire  which  was  administered  at  the  conclusion  of  the  evaluation.  The  objective 
performance  measures  consisted  of  the  number  of  GCAs  completed,  path  score,  gate  score,  tot"!  score,  glide 
slope  error,  course  angle  error,  and  angle  of  attack  error.  These  measures  were  printed  out  at  the 
termination  of  each  GCA.  Appendix  A presents  a description  of  these  measures. 
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The  path  score  was  the  weighted  average  of  the  percentage  of  time  the  pilot  flew  the  aircraft  at  the 
prescribed  glide  slope  angle,  course  heading,  and  angle  of  attack.  These  measures  were  taken  from  the  time 
the  aircraft  intersected  the  glideslope  until  it  reached  decision  height,  the  point  at  which  the  landing  could 
or  could  not  be  completed.  Decision  height  was  defined  as  200  feci  of  altitude  and  normally  occurred  at 
approximately  .75  miles  from  touchdown.  A missed  approach  was  commanded  when  the  aircraft  deviated 
from  the  simulated  normal  GCA  approach  cone  for  Runway  21  at  Luke  AFB. 

The  gate  score  essentially  was  a “snapshot”  look  at  how  well  the  aircraft  was  set  up  for  landing  at  the 
decision  height.  In  addition  to  information  about  deviations  from  desired  glide  slope  angle,  course  heading 
and  angle  of  attack,  this  score  also  reflected  heading  rate  and  altitude  rate  information. 

The  total  score  was  computed  by  adding  the  gate  score  and  the  path  score  together  and  adding  one 
hundred  points.  Within  the  adaptive  software  program,  the  total  score  was  used  to  determine  the  rate  of 
increase  or  decrease  in  difficulty  level  for  a pilot. 

for  purposes  of  the  evaluation,  the  AFTS  software  was  modified  to  furnished  roo! -mean-square 
(RMS)  glide  slope  angle  error,  RMS  course  angle  error,  and  RMS  angle  of  attack  error.  Measurement  for 
these  scores  was  initiated  when  the  aircraft  reached  eight  miles  and  was  terminated  at  decision  height  or  at 
the  missed  approach  point 

GCA  Training  in  the  Normal  TAC  F-4  Training  Program 

The  norma!  F-4  simulator  syllabus  consists  of  22  sorties,  each  lasting  approximately  1.5  hours.  The 
c’  crage  student  receives  11  simulator  GCAs  of  various  types  during  F4  training.  The  majority  of  these 
GCAs  are  given  during  the  first  eight  simulator  sorties. 

Experimental  Design 

The  present  evaluation  was  performed  within  the  framework  of  the  normal  F4  syllabus  with  minor 
modifications.  The  experimental  design  for  this  study  is  presented  in  Table  1.  All  O'" A training  was 
accomplished  during  the  first  fifteen  sorties.  Students  received  one  GCA  on  each  of  the  first  two  simulator 
sorties  and  three  GCAs  on  each  of  the  remaining  sorties  up  to  and  including  Sortie  #15.  Equipment 
problems  forced  cancellation  of  one  sortie  consisting  of  three  GCAs.  Thus,  a total  of  38  GCAs  were  given 
during  the  training  sorties. 

The  24  student  pilots  from  F-4  Class  75-CRL  were  assigned  randomly  to  two  equal  groups.  Twelve 
students  receive*.  GCA  training  using  the  AFTS.  The  remaining  12  students  received  standard  GCA  training 
administered  by  instructor  pilots  and  simulator  operators. 

To  assure  equivalent  training  between  the  two  groups,  students  were  randomly  paired  so  that  one 
member  of  the  pair  was  assigned  to  the  AFTS  GCA  Training  Group  and  the  other  member  of  the  pair  was 
assigned  to  the  Standard  GCA  Training  Group.  The  pair-member  assigned  to  the  AFTS  Group  always 
received  GCA  training  before  his  counterpart.  The  pair-member  assigned  to  the  Standard  Group  received 
the  same  order  of  presentation  and  difficulty  levels  of  GCA  training  tasks  as  his  AFTS  trained  counterpart. 
This  yoked  experimental  design  was  used  in  order  to  avoid  the  potential  criticism  of  unequal  training.  An 
overall  feature  of  the  design  was  that  the  training  for  the  two  groups  was  equal  in  terms  of  the  sequence  of 
presentation  and  the  specific  properties  of  the  GCA  problems  presented. 

Criterion  sorties  consisting  of  six  GCAs  each  were  administered  to  all  students  at  the  completion  of 
Simulator  Sorties  #15  and  #22.  For  each  criterion  sortie,  three  GCAs  representing  different  difficulty  levels 
were  given  by  the  AFTS.  The  same  three  GCAs  also  given  by  highly  qualified  professional  GCA  controllers, 
thereby  giving  each  student  a total  of  six  GCAs. 

Procedure 

Logicon  GCA  Training  Group.  The  AFTS  provided  training  for  ASR  and  PAR  approaches.  The 
system  was  designed  for  training  using  the  instructor  console  of  the  AFTS.  The  pilot  was  directed  by  the 
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Table  I.  Experimental  Design  for  the  AFTS  Evaluation 


Sortie 

Adaptive  Group 
(AFTS  T raining) 
N = 12 

Control  Group 
(Normal  Training) 
N = 1 2 

Training  Days 

l 

1 A through  1 2A 

IB  through  12B 

6,7 

1 GCA  per  student 

1 GCA  per  student 

T 

1 GCA  per  student 

1 GCA  per  student 

8.6 

3 

3 GCA’s  per  student 

3 GCA’s  per  student 

t0, 11,  12 

4 

3 GCA’s  per  student 

3 GCA’s  per  student 

13, 14,  15 

5 

3 GCA’s  per  student 

3 GCA's  per  student 

16, 17, 18 

6 

3 GCA's  per  student 

3 GCA’s  per  student 

19,20,21 

7 

3 GCA’s  per  student 

3 GCA's  per  student 

TO  T ^ 14 

8 

3 GCA’s  per  student 

3 GCA's  per  student 

26,27,28 

9 

3 GCA’s  per  student 

3 GCA's  per  student 

29.30.31 

10 

3 GCA’s  per  student 

3 GCA’s  per  student 

32,33.34 

1 1 

3 GCA’s  per  student 

3 GCA's  per  student 

35.36,37 

12 

3 GCA’s  per  student 

3 GCA’s  per  student 

38,39.40 

13 

3 GCA’s  per  student 

3 GCA's  per  student 

41.42.43 

14 

3 GCA's  per  student 

3 GCA’s  per  student 

44,45,46 

15 

3 GCA’s  per  student 

3 GCA’s  per  student 

53,54.55 

Criterion 

Criterion  Test 

Criterion  Test 

61.62.63. 

Ride 

3 AFTS  GCA's 

3 AFTS  GCA's 

64 

3 GCA  Controller  GCA's 

3 GCA  Controller  GCA’s 

16-22 

No  GCA’s 

No  GCA's 

Criterion 

Criteiron  Test 

Critcion  Test 

112.  1 13,114. 

Ride 

3 AFTS  GCA’s 

3 AITS  GCA’s 

115 

3 GCA  Controller  GCA’s 

3 GCA  Controller  GCA’s 

automated  voice  to  contact  the  Phoenix  approach  controller,  establish  an  altitude  of  3.000  feet  and  a 
heading  of  210  degrees.  When  these  conditions  were  satisfied  and  the  software  program  had  adjusted  the 
aircraft  weight,  wind  direction  and  velocity,  and  atmospheric  turbulence,  then  the  aircraft  was  positioned 
to  12  mile:  from  touchdown.  At  a distance  of  10  miles  from  touchdown,  die  pilot  was  instructed  to  make  a 
radio  frequency  change  and  to  contact  the  final  approach  countroller.  The  final  controller  “talked"  the 
pilot  down  the  glidepath  with  information  about  the  gildepath,  glidcslopc,  and  range  from  touchdown.  The 
ASR  approach  was  similar  except  that  gildepath  information  was  not  given.  Aircraft  emergencies  were 
introduced  at  ranges  of  10  miles  to  2 miles  from  touchdown.  Students  in  this  training  group  were  permitted 
to  examine  the  training  records  printed  out  by  the  AITS  and  to  use  the  playback  system  at  any  time  during 
the  study. 

Standard  GCA  Training  Group.  The  procedure  for  students  in  this  group  was  similar  to  the  AFTS 
trained  group.  The  AFTS  was  used  to  set  up  the  problem  and  to  perform  the  performance  measurement. 
However,  after  the  pilot  contacted  the  final  GCA  controller,  the  automated  voice  was  locked  out  and  the 
instructor  pilot  or  console  operator  performed  the  task  of  “talking"  the  student  down  the  gildepath. 

The  Situation  Display  Indicator  (SDI)  on  the  simulator  main  console  was  programmed  to  present 
course  and  glideslopc  information  to  the  individual  who  served  as  the  GCA  controller.  The  GCA 
presentation  on  the  SDI  was  displayed  in  a manner  similar  to  a normal  radar  scope  GCA  presentation. 

Students  in  this  training  group  were  not  allowed  to  see  their  performance  sheets  or  to  use  the 
playback  feature  of  the  AFTS.  Instructors  were  not  permitted  to  use  any  part  of  the  AFTS  for  debriefing 
or  feedback  to  the  student.  In  short,  the  only  feedback  that  students  in  this  group  received  was  from 
standard  sources  available  in  F-4  training. 
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Criterion  Test.  Approximately  four  months  elapsed  between  the  two  criterion  rides.  During  this 
interval,  students  did  not  receive  further  GCA  training  in  the  simulator;  however,  students  did  report 
receiving  one  or  two  GC,is  in  the  aircraft. 


On  each  criterion  sortie,  the  student  was  briefed  on  the  procedure  to  be  followed  and  the  conditions 
that  were  to  be  given  for  each  difficulty  level.  Three  GCAs  were  given,  then  the  student  was  given  a five 
minute  rest  period;  then  the  last  three  GCAs  were  administered. 


The  same  difficulty  levels  were  administered  during  the  first  and  second  criterion  sorties.  The  order  of 
presentation  of  the  difficulty  levels  was  also  the  same  for  both  sorties.  Difficulty  levels  selected  were:  (1) 
Level  1,  Wind  2 ! 0 degrees  (head  wind)  at  35  knots,  aircraft  weight  35,000  pounds,  no  turbulence,  no 
emergency ; (2)  Level  30,  Wind  030  degrees  (tail  wind)  at  20  knots,  aircraft  weight  43,700  pounds,  heavy 
turbulence,  no  emergency;  and  (3)  Level  49,  Wind  030  degrees  (tail  wind)  at  20  knots,  aircraft  weight 
43,700  pounds,  heavy  turbulence,  left  engine  failure  at  eight  miles  from  touchdown. 


During  the  first  criterion  sorties,  half  the  students  in  each  training  group  were  given  the  AFTS  GCAs 
first  while  the  other  half  of  the  students  in  that  group  received  GCAs  given  by  the  professional  controllers 
first.  During  the  second  criicrion  sortie,  the  order  of  presentation  for  GCAs  was  reversed. 


ID.  RESULTS  AND  DISCUSSION 


The  results  of  the  evaluation  are  presented  and  discussed  according  to  four  topical  areas:  (1)  the 
training  effectiveness  of  the  system,  (2)  qualitative  evaluation  of  the  system,  (3)  operational  reliability  and 
maintainability  of  the  system,  and  (4)  recommended  changes  and  operational  utilization  of  the  system. 


Evaluation  of  Training  Effectiveness 

An  examination  of  differences  between  performance  data  for  the  adaptive  (AFTS-trained)  and 


control  (instructor-trained)  groups  provides  information  regarding  the  training  effectiveness  of  the  AFTS 
system.  If  the  AFTS  was  to  be  considered  an  cfiective  training  system,  then  students  in  the  adaptive  group 
should  have  performed  better  or  at  least  as  well  as  those  stuJents  in  the  control  group.  Otherwise,  the 
operational  utility  of  such  a system  would  be  seriously  questioned.  Data  pertaining  to  this  question  could 
be  divided  according  to  source  - that  obtained  during  the  training  period  and  that  obtained  during  the  two 
criterion  sorties. 


Training  Data.  The  original  experimental  design  called  for  one  GCA  to  be  administered  on  the  first 
two  sorties  and  three  GCAs  on  the  remaining  thirteen.  Due  to  system  failure,  GCAs  for  one  sortie  were 
cancelled,  thereby  reducing  the  total  number  to  38.  Of  these,  six  were  emergencies  required  by  the  training 
syllabus  on  specified  sorties.  Therefore,  only  32  GCAs  were  administered  as  originally  intended  for  each 
student.  Furthermore,  one  student  in  the  control  group  suffered  a fractured  collarbone  and  was  placed  on  a 
medical-hold  status,  thereby  eliminating  him  from  the  study. 


Since  both  the  adaptive  and  control  groups  received  the  same  levels  of  difficulty,  inferences  were 
based  on  performance  data  rather  than  measures  derived  from  difficulty  levels.  Nonetheless,  data  reflecting 
changes  in  difficulty  level  as  a function  of  training  trials  are  presented  for  descriptive  purposes.  Figure  1 
presents  the  mean  difficulty  levels  for  the  adaptive  group.  The  data  are  characterized  by  increasing  means 
and  variability  as  a function  of  training  trials  and  are  presented  in  Appendix  B.  Plots  for  each  individual  are 
also  presented  in  Appendix  B. 


An  examination  of  these  individual  learning  curves  reveals  certain  trends.  It  appears  that  the  major 
difference  among  individuals  is  not  the  siope  of  the  learning  curve,  but  rather  the  number  of  trials  before 
consistent  advancement  in  difficulty  level  occurs.  In  other  words,  once  the  student  has  mastered  the 
concept  of  the  CCA,  and  what  the  requirements  are,  then  he  will  consistently  advance.  Otherwise,  he 
remains  at  the  lower  levels  of  difficulty.  For  such  students  who  do  not  master  the  GCA  concept  quickly,  it 
is  unlikely  that  the  introduction  of  emergency  conditions  will  be  of  any  value.  The  individual  learning 
curves  for  several  student0  indicate  this  to  be  the  case. 


The  most  important  question  to  be  addressed  by  the  training  data  concerned  potential  differences  in 
performance  for  the  adaptive  and  control  groups.  Data  for  the  32  GCA  training  trials  were  combined  to 
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Figure  I.  Mean  difficulty  level  as  a function  of  training  trials. 


produce  1 training  blocks.  The  first  block  consisted  of  the  first  two  GCAs  given  on  the  first  two  sorties. 
The  remaining  ten  blocks  consisted  of  consecutive  sets  of  three  training  trials.  Tor  each  performance 
measure,  the  block  score  was  simply  the  mean  of  all  trial  scores  within  that  block.  Seven  measures  of 
performance  were  used  path  completion  (scored  0 or  1),  glide  slope  score,  course  angle  score,  angle  of 
attack  score,  path  score,  gate  score,  and  total  score.  Descriptive  statistics  were  computed  for  these  data  and 
the  results  are  presented  in  Appendix  B.  Mean  scores  for  each  measure  pooled  across  all  training  blocks  arc- 
presented  in  Table  2. 


Hach  dependent  measure  was  analyzed  by  a split-plot  factorial  analysis  of  variance  (Kirk,  1968) 
having  one  between  factor  (adaptive  vs  control)  and  one  within  factor  (training  block).  These  results  are 
presented  in  Table  3.  As  indicated,  only  the  training  block  factors  produced  a significant  effect  for  the 
dependent  measures.  Neither  the  group  factor  nor  its  interaction  with  the  training  block  factor  was 
significant.  In  other  words,  the  data  revealed  no  difference  in  performance  between  the  adaptive  and 
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Tabic  2.  Mean  Scores  for  Training 
Sortie  Data 


Measure 

Adaptive 

Control 

Path  Completin 

.613 

.606 

Glideslopc  Score 

58.01 1 

59.877 

Course  Angle  Score 

64.007 

66.359 

Angle  of  Attack  Score 

30.279 

28.277 

Path  Score 

57.806 

58.481 

Gate  Score 

25.445 

26.695 

Total  Score 

165.158 

162.663 

Tabic  Summary  of  Analyses  of  Variance 
for  Training  Data 


Measure 

A 

B 

AB 

Path  Completion 

.0058 

8.1400** 

.3290 

Glideslopc  Score 

.7018 

3.8310** 

.7127 

Course  Angle  Score 

.8727 

2.1881* 

.7268 

Angle  of  Attack  Score 

1915 

2.9454** 

1.0656 

Path  Score 

.1426 

8.2004** 

.8340 

Gate  Score 

.1071 

5.3284** 

.3622 

Total  Score 

.0322 

10.7876** 

.3117 

*p<.O.S  A (troup  Factor 

■‘p^.Ol  11  Trial  Block  Factor 


control  groups.  A priori  t tests  were  computed  to  determine  whether  ar.y  group  differences  existed  during 
die  first  training  block.  Again,  no  differences  were  found  suggesting  the  initial  ability  levels  Tor  the  two 
groups  to  be  equivalent. 

Since  no  group  differences  were  indicated  the  data  were  pooled.  Figure  2 presents  mean  percent  of 
completed  GCAs  completed  across  the  1 1 training  blocks.  Figures  3,4,  and  5 present  each  or  the  remaining 
dependent  measures  as  a function  of  training  block  number.  The  data  for  these  dependent  measures  arc 
consistent.  There  is  an  initial  increase  in  performance  through  the  First  training  blocks  followed  by  a 
decrease  and  another  increase.  It  seems  likely  that  the  decrease  noted  in  blocks  5 through  9 reflect  the 
introduction  of  difficulty  levels  requiring  emergencies.  In  any  case,  it  is  apparent  that  measures  of 
performance  do  change  as  a function  of  training  trials.  The  adaptive  logic  docs  not  alter  difficulty  level  so  as 
to  maintain  the  performance  data  constant. 

To  summarize,  the  data  reflect  no  differences  between  the  adaptive  and  control  groups  during  the 
training  period.  An  examination  of  the  descriptive  statistics  reported  in  Appendix  B reveal  the  results  to  be 
nearly  identical.  It  seems  safe  to  conclude  that  the  data  indicates  both  groups  received  equivalent  training. 
No  differences  could  be  detected. 

Criterion  Sortie  Data.  For  each  criterion  sortie,  half  of  the  GCAs  were  controlled  by  the  machine 
while  the  other  half  were  administered  by  highlv  qualified  GCA  controllers.  Performance  under  the 
experienced  GCA  controllers  was  assumed  to  represent  the  major  criterion  for  evaluating  the  training 
effectiveness  of  the  AFTS  ystem.  Simply  stated,  were  there  performance  differences  between  the  adaptive 
and  control  groups  using  experienced  controllers1  The  primary  concern  was  whether  the  adaptive  group, 
trained  on  the  machine,  could  effectively  transfer  to  the  GCA  task  using  actual  controllers.  Aside  from  this 
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BLOCKS  OF  TRIALS 

I' idlin'  5.  Mean  total  scores  as  a function  of  training  blocks. 

question,  the  data  permitted  an  evaluation  of  the  automated  machine  eontrollci  using  the  experienced  (i('A 
controller  as  the  standard. 

To  answer  these  questions,  an  analysis  of  variance  was  performed  lor  each  dependent  measure.  T he 
factorial  design  consisted  of  one  between  subjects  factoi  (group)  and  three  within  subjects 
factors  criterion  sortie  (1st  vs.  2nd),  type  of  controller  (machine  vs.  human  operator),  and  level  of 
difficulty  (Level  i vs.  Level  TO  vs.  Level  40).  A summary  of  the  resulting  I* -values  for  all  main  effects  and 
interactions  are  presented  for  each  dependent  measure  in  Table  4. 

As  the  data  indicate,  the  only  significant  main  effects  were  those  factors  reflecting  type  of  controller 
and  level  of  difficulty.  Professional  GCA  controllers  produced  significantly  better  scores  for  all  measures 
except  GCA  completion.  RMS  angle  of  attack,  and  the  gate  score.  The  group  means  are  presented  in  T able 
5.  Tor  the  levels  of  difficulty  factor,  only  the  RMS  angle  of  attack  measure  was  not  significant,  (iroup 
means  for  this  factor  are  presented  in  Table  (>.  The  data  indicate  that  measured  performance  for  Levels  I 
and  30  to  be  roughly  the  same,  while  significantly  degraded  for  Ixvel  40.  the  single-engine  emergency.  The 
exception  is  the  path  score  measure  wherein  Levels  I and  40  are  equivalent  with  Level  30  yielded  the  better 
performance.  However,  it  must  be  recalled  that  Levels  30  and  40  were  Down  under  conditions  of  maximum 
turbulence.  The  measurement  alogrithm  adds  15  points  to  the  path  score  to  compensate  for  such 
turbulence.  Consequently,  subtracting  this  amount  from  the  mean  reveals  Levels  I and  30  to  be  roughly 
equivalent,  with  performance  on  level  40  significantly  degraded.  Since  the  path  score  is  part  of  the  total 
score,  the  same  logic  applies.  In  summary,  the  criterion  data  suggests  performance  on  Levels  I and  d(l  tube 
roughly  equivalent.  Performance  is  significantly  degraded  only  on  Level  40,  the  single  engine  emergency. 

As  previously  indicated,  the  critical  comparisons  were  between  the  two  groups  for  the  GCAs 
conducted  by  the  actual  controllers.  Table  7 presents  the  mean  values  for  each  of  the  dependent  measures. 
A priori  t-tests  were  computed  lor  each  measure.  No  statistical  differences  were  obtained.  In  other  words. 
the  adaptive  group  trained  by  the  machine  controller  performed  as  well  with  the  actual  (ICA  controllers  as 
did  those  trained  by  human  operators. 
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Table  4.  Summary  of  Analyses  of  Variance  for  Criterion  Ride  Data 


Source 

RMS 

Glide 

Slope 

RMS 

Course 

Angle 

RMS 
Angle  of 
Attack 

Path 

Score 

Gate 

Score 

Total 

Score 

A 

.02(i 

.300 

.137 

.053 

1 .308 

.688 

B 

.639 

.168 

3.820 

.981 

.613 

1.046 

C 

10.325** 

24.707** 

.804 

40.800** 

2.199 

5.554* 

D 

26.284** 

30.517** 

1.078 

21.322* 

12.880** 

1 1.406** 

All 

.068 

.000 

.169 

.060 

1 .320 

.423 

AC 

.788 

.155 

1.314 

1.458 

.914 

1.178 

Al) 

.062 

.171 

.503 

.175 

1.028 

.293 

BC 

.010 

.1  il 

.002 

.209 

.314 

.025 

111) 

.466 

1.504 

.458 

1.415 

.336 

.019 

CD 

1.258 

1.077 

.924 

.286 

.143 

.038 

ABC 

» 7 8 

2.810 

.008 

2.736 

.060 

.281 

A 111) 

1.414 

.324 

1.539 

.129 

.192 

.024 

ACD 

.407 

2.97! 

1.401 

.877 

.288 

1.548 

BCD 

122 

2.249 

2.431 

.378 

4.485* 

4.560* 

ABCD 

.514 

.082 

.896 

.35o 

4.192* 

3.272* 

•p  <.ns 
’•p  <.oi 


(■roup 

(YiPrion  Sort  if 
Tv  pc  of  Controller 
Level  ‘if  Difficult) 


Table  5.  Mean  Scores  as  a Function 
of  Type  of  Controller 


Measure 

GCA  Operator 

Machine 

Path  Completion 

.833 

.812' 

RMS  Glideslopc 

.241 

294* 

RMS  Connie  Angle 

.320 

.454* 

RMS  Angle  of  Attack 

1.885 

1.820 

Path  Score 

80.141 

71.387* 

Gate  Score 

46.627 

42.629 

Total  Score 

220.126 

207.472* 

N i » Significant  r I * 

* Signifu  .mt  Hi  f termers 

Table  b.  Mean  Scores  as  a Function 
of  Levels  of  Difficulty 


Measure 

Level  1 

Level  30 

Level  49 

Path  Completion 

.935 

.870 

.663' 

RMS  Glideslopc 

.214 

.208 

.380* 

RMS  Course  Angle 

.284 

.323 

.553* 

RMS  Angle  of  Attack 

1 .737 

1 .834 

1.991 

Path  Score 

71.078 

85.386 

70.838* 

Gate  Score 

52.882 

48.439 

32.650* 

Total  Score 

221.448 

226.794 

193.157* 

lNo  Si^nifit ante  Test 
* Significant  nilfrrrncrs 
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Table  7.  Mean  ( •roup  Scores  for  GCAs 
Administered  hv  Professional  Controllers 


Measure 

Adaptive 

Control 

I’ath  Completion 

.Kb  1 

.SO  7 

RMS  Gildeslope 

,27b 

.247 

RMS  Course  Angle 

.770 

.200 

RMS  Angle  of  Attack 

1 .SO  5 

1 ,S74 

I’at  h Score 

7b  ,07b 

75. OS  1 

Gate  Score 

-1S.25  1 

44.074 

Total  Score 

221.412 

2 IS. 7 27 

Ilv  only  significant  interactions  involving:  a group  effort  were  fur  the  gale  scute  and  total  score 
measure;  An  analysis  of  the  fouitli  oidet  inteiaclion  lot  the  gate  score  tevealed  that  the  adaptively  trained 
group  performed  significantly  better  on  Level  70  during  the  fust  auction  sortie  in  which  the  machine 
saved  as  the  controller.  The  total  score  produced  similar  findings.  1 -tests  between  the  groups  for  each 
difficulty  level  revealed  no  differences  whenever  the  human  (it  A controllers  were  providing  commentary. 

Considering  the  data  collected  during  the  two  auction  ride  sorties,  it  is  apparent  that  no  reliable 
differences  m performance  could  be  delected  between  the  adaptive  and  control  groups.  However,  tlic  trends 
o)  the  criterion  test  data  rejlei  tint:  somewhat  heller /nrjornwnec  of  the  adaptively  trained  students  and  the 
high  total  seores  for  both  groups  lead  to  the  eonelusinn  that  the  AI-TS  is  an  effective  system  for  training 
(iCAs.  The  machine  appears  to  tram  as  well  as  the  instructor  and  apparently  does  not  truin  anv  adverse 
CCA  responses  as  measured  during  this  study.  Although  the  data  have  clearly  established  ALT'S  to  be  an 
effective  training  device,  the  cost-effectiveness  ol  the  system  lemains  a question  beyond  the  scope  of  the 
present  evaluation. 

Qualitative  Evaluation  of  the  AITS 

At  the  completion  ol  the  final  criterion  sortie,  a questionnaire  was  developed  to  furnish  qualitative 
information  about  the  All  S . It  was  admimsteied  to  students,  instructors,  and  GCA  controllers  who 
participated  in  the  evaluation.  Section  I of  the  questionnaite  examined  specific  operating  characteristics  of 
the  AI  TS  and  recommendations  lor  change.  Section  II  dealt  with  how  the  system  was  used  and  how  its  use 
could  be  improved.  Comments  loi  all  gioups  ol  respondents  have  been  summarized  and  are  presented  as 
Initows' 

Section  I.  I.ist  things  liked,  disliked,  and  recommended  changes  for  each  of  the  following  phases  of 
M TS  svstem  operation. 

■I.  Set-up  to  final  (1C  A controller 

Students  and  instructors  pointed  out  that  when  a si  tong  crosswind  was  pieient.  the  automated 
contiollei  did  not  have  the  pilot  make  a wind  correction  until  the  final  approach.  This  was  often  too  late  to 
take  corrective  action  and  resulted  in  a pool  GCA  oi  a missed  approach,  flic  GCA  controllers  indicated  '.hat 
a good  controller  would  pick  up  the  inbound  airmail  much  earliei  il  course  corrections  were  required. 
Another  complaint  ol  the  contiolleis  was  that  the  ALT'S  does  not  give  the  pilot  enough  corrective 
inlormation  early  in  the  approach.. 

Students  stated  that  it  multiple  appmaches  aie  to  be  perlotmed  at  difleient  difficulty  levels,  mote 
tune  should  be  allowed  alter  changmt.  the  airmail  gross  weight  and  configuration  so  that  the  aircraft  can  be 
trimmed  before  star'ing  the  approach.  Anothei  major  oh|cctum  about  the  set-up  configuration  ol  the  Al  TS 
was  that  the  sy  stem  initially  instructed  the  pilot  to  climb  to  an  altitude  ol  5.000  feet  and  a heading  ol  210 
degiees.  I pon  reaching  the  assigned  altitude  and  heading,  the  ALTS  then  instructed  the  pilot  to  descend 
and  maintain  7.0(H)  leet  and  a heading  ol  210  degrees.  There  is  no  apparent  need  to  climb  to  a higher 
altitude  than  7.000  'eel . 
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/?.  Final  anil  roller  operation  ( Voiee /corrections) 

Students  and  instructors  indicated  that  AFTS  voice  commands  were  too  slow.  Comments  from  both 


groups  stated  that  the  machine  was  unlike  an  actual  controller  in  that  it  was  unable  to  adjust  its  instruction 
rate  to  different  pilot  correction  rales.  Instructors  stated  that  the  apparent  lag  in  instructions  required  the 
student  to  anticipate  control  responses.  Examination  of  the  voice  generation  system  for  the  AFTS  show 
that  this  criticism  will  be  difficult  to  overcome  with  the  present  system.  The  Metrolab  voice  system  uses 
fixed  length  half  second  phrases;  thus,  the  oidy  method  of  increasing  the  number  of  words  delivered  in  a 
fixed  length  phrase  would  be  to  put  more  words  on  the  half  second  phrase  unit. 


A frequent  complaint  about  the  AFTS  was  that  the  priorities  of  voice  corrections  need  to  be  changed 
or  reexamined.  Tire  priority  of  azimuth  and  elevation  corrections  were  difficult  for  the  machine  to  give 
simultaneously  Often,  the  AITS  would  give  instructions  for  correcting  one  type  of  error  while  ignoring 
another.  Students  and  instructors  indicated  that  at  longer  ranges  from  touchdown,  pilots  need  heading 
information;  as  the  aircraft  nears  touchdown,  glideslopo  information  becomes  more  important.  GCA 
controllers  stated  that  priorities  on  trend  information  are  unrealistic.  Corrective  information  should  be 
given  first  and  then  information  about  error  is  given.  The  controllers  suggested  that  priorities  should  be  to 
give  information  dealing  with  the  features  of  the  approach  in  greatest  error;  c.g.,  if  the  aircraft  is  on  proper 
glide  slope  but  going  off  on  heading,  then  the  AFTS  should  give  the  heading  information  prior  to  the  glide 
slope  information. 


Students,  IPs,  and  controllers  indicated  that  the  angle  of  attack  aural  tone  is  cut  out  when  an 
instructor  or  a real  GCA  controller  is  giving  instructions  over  the  communications  system.  Since  students 
and  pilots  use  the  aural  tone  to  maintain  the  desired  angle  of  attack,  this  problem  should  be  examined  and 
modifications  made  to  the  simulator  communications  system  if  appropriate. 

Students  and  instructors  commented  that  the  AFTS  controller  appeared  to  give  corrections  when  no 
corrections  were  required.  For  example,  if  the  aircraft  was  steady  at  210  degrees,  the  AFTS  would 
command  “On  Course.  Turn  left  20X"  This  message  often  was  followed  by  the  instruction  “Left  of  course, 
turn  right  to  a heading  of  210  degrees."  Heading  information  was  giver,  too  often  with  many  one  degree 
corrections.  This  problem  can  be  solved  by  changing  the  AFTS  error  tolerances  to  realistic  levels. 

C.  Performance  measurement  (print-out) 

Students,  instructors,  and  controllers  stated  that  this  feature  of  the  system  was  excellent.  Students 
suggested  that  the  system  should  give  a grade  fur  smoothness  and  consistency  of  performance.  They  also 
indicated  that  angle  of  attack  for  emergencies  such  as  single  engine  failures  should  be  modified  so  that 
students  are  not  downgraded  tor  flying  a different  angle  of  attack  if  necessary.  Although  students  found  the 
printout  to  be  an  excellent  motivational  device,  they  commented  that  the  printout  should  be  better 
explained  and  critiqued  so  that  students  and  instructors  know  how  to  use  the  information  provided. 

Instructor  comments  indicated  that  although  they  considered  the  printout  to  be  a valuable  tool  for 
debriefing  the  student,  several  improvements  should  be  considered.  The  gate  score  appears  to  receive  too 
much  weight  in  the  performance  score  of  the  student.  It  was  suggested  that  the  scoring  at  missed  approach 
should  include  the  number  of  feet  ieft/right  and  above/below  the  required  decision  height.  This  infomiation 
would  allow  the  student  and  instructor  to  evaluate  the  approach  more  effectively  than  the  present 
information  permits. 

Although  it  was  labeled  as  a minor  comment,  several  instructors  stated  that  the  present  printout  tends 
to  waste  paper  with  only  one  printout  per  sheet.  They  suggested  the  use  of  a CRT  read  out  with  the 
capability  of  getting  hardcopy  is  desired. 

I).  Replay  Capability 

All  students  and  instructors  who  participated  in  the  evaluation  and  who  were  exposed  to  the  replay 
feature  of  the  AFTS  commented  that  this  was  the  outstanding  feature  of  the  system.  Although  students  in 
the  control  group  were  not  exposed  to  the  replay  feature,  several  students  from  this  group  commented  that 
they  would  have  liked  such  a feature. 

During  the  evaluation,  an  attempt  was  made  to  measure  the  amount  of  use  of  the  replay  system  by 
having  aircrews  sign  in  and  out  when  they  used  it  However,  this  method  did  not  prove  to  be  effective  and 
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(lie  information  was  requested  in  the  questionnaire.  As  might  he  expected,  student  lesponses  ranged  from 
"infrequently"  to  “often."  The  majority  of  students  indicated  that  they  used  the  system  as  much  as 
possible  because  it  permitted  them  to  see  what  had  been  performed  correctly  and  what  errors  had  been 
made. 


Students  commented  that  the  replay  system  frequently  was  unavailable  for  use  because  it  was  located 
in  a standard  briefing  room  which  was  used  by  other  aircrews.  Another  piohlem  in  using  the  replay  feature 
was  the  result  of  the  long  training  days  which  were  sometimes  experienced '.  students  and  instructors  simply 
were  too  tired  or  too  busy  to  use  the  system  at  the  moment  they  completed  their  simulator  training. 
Several  students  and  instructors  suggested  the  possibility  of  placing  the  replay  lei  urinal  in  the  squadron  or 
other  more  convenient  location. 


A frequent  comment  from  students  and  instructors  was  that  the  replay  system  should  be  made  easier 
to  use  by  non-programmers,  lire  present  replay  system  required  the  user  to  enter  several  abstract 
programming  instructions  to  the  machine  in  older  to  use  the  replay.  This  criticism  could  be  satisfied  by 
developing  a human  engineered  fonuat  which  would  lead  the  user  through  the  correct  sequence  of 
operations.  Ideally,  the  only  specific  knowledge  that  should  be  required  of  the  user  is  how  to  turn  the 
replay  system  on  ami  off 


/■•'.  Task  sequencing  I changes  in  level  ofJiJIiculty) 

Students.  IPs.  and  (i('A  controllers  indicated  that  the  task  sequencing  of  the  APTS  is  good  in  that  it 
permit!‘’d  the  students  to  progress  at  a rale  commensurate  with  their  skill  level.  Controllers  stated  that  a 35 
knot  head  wind  is  not  realistic.  They  suggested  that  a better  training  difficulty  piohlem  would  be  to  use  a 
l,0  degiee  crosswind  or  gusting  winds  of  some  type.  Although  control  group  students  were  not  exposed  to 
the  automated  task  sequencing,  most  students  in  this  group  indicated  that  they  would  have  liked  this  type 
of  training. 

Section  //.  Discuss  each  <>)  the  /allowing  questions: 

A.  How  realistic  is  the  system'1 


highly -two  percent  of  the  students  considered  the  AITS  a realistic  system  for  giving  C1C  A training. 
Instructors  stated  that  compared  to  the  realism  of  the  P-4  simulator,  the  APTS  is  the  most  realistic  portion 
of  the  training  received  in  the  simulator.  Instructors  and  GCA  controllers  suggested  that  the  realism  of  the 
APTS  can  be  enhanced  by  adjustments  in  the  priorities  of  the  messages  and  increases  in  the  rate  of 
information  from  the  automated  voice. 


II.  How  effective  is  the  system  in  terms  of  enabling  you  to  fly  (iCAs  in  the  Aircraft ? 

Pighty-nine  percent  of  the  students  stated  that  the  APTS  provided  good  instrument  cross-check 
training  and  that  the  training  was  effective  for  them.  Instructors  supported  the  value  of  the  system  and 
indicated  that  the  system  gives  better  GCA  training  than  normal  IP  training  because  of  the  standardized 
program.  IPs  stated  that  they  do  not  get  enough  practice  giving  GCAs  to  become  proficient  in  providing 
GCA  training.  Another  factor  identified  was  that  Puke  APB  is  a fair  weather  base;  therefore,  students 
receive  very  few  (iCAs  in  the  aircraft  Several  instructors  said  that  it  might  be  a good  idea  to  bring  in  real 
GCA  controllers  periodically  to  give  training  to  students,  but  still  use  liic  APTS  measurement  system. 

C.  What  is  the  most  effective  way  to  utilize  the  system  in  operational  training'’ 


Student  opinion  for  this  question  jS equally  divided.  Responses  indicated  that  the  system  should  be 
used  in  an  individualized  instruction  n.o-  c r.r  as  the  study  used  the  system  with  a specified  number  of 
GCAs  required  for  each  simulator  mission.  The  majority  of  the  students  indicated  that  they  would  like  to 
see  the  objective  scoring  system  applied  to  air  attack,  low  level  radar  attack,  and  radar  land  mass  training. 
Instructors  concurred  with  the  comments  of  the  students  and  suggested  that  the  simulator  equipped  with 
A ITS  be  the  only  training  system  used  to  train  GCAs.  The  majority  of  the  instructors  strongly  urged  that 
the  GCA  training  should  require  the  proper  radio  calls  by  the  aircrews.  Instructors  pointed  out  that  the 
‘■.P  IS  could  be  used  to  decrease  the  number  of  aircraft  GCAs  that  TAC  Manual  51-34  requires  every  six 
months.  Most  instructors  believed  that  students  should  be  permitted  to  schedule  GCA  training,  but  bring 
the  Al  l S grade  sheet  to  the  instructor  for  signoff  that  training  has  been  accomplished. 
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D.  What  should  he  the  role  of  the  instructor  in  training  GCAs  using  the  AFTS? 

The  majority  of  students  stated  that  an  instructor  is  not  required  for  GCA  training  using  the  AFTS. 
Students  and  instructors  stated  that  the  ;olc  of  the  instructor  should  be  to  judge  the  learning  trend  of 
students,  critique  student  performance,  monitor  the  AFTS,  and  to  help  the  student  correct  errors.  Several 
students  suggested  that  instructors  should  give  an  occasional  GCA  so  that  the  student  does  not  get  used  to 
the  same  voice,  phrases,  and  correction  rates. 

A . What  is  the  one  thing  you  liked  best  about  l.ugicon  and  what  is  the  one  thing  you  disliked  most  ? 

Students  identified  the  following  features  of  the  AFTS  as  the  tilings  they  liked  about  the  system:  (1) 
printout  of  performance;  (2)  replay  capability;  (3)  standardization  of  training;  (4)  capability  to  use  AFTS 
without  reliance  upon  instructors;  (5)  the  challenge  of  trying  to  beat  the  machine;  and  (6)  the  variety  of 
GCAs  using  different  winds,  aircraft  weights,  turbulence,  and  emergencies.  Instructors  stated  that  the 
features  of  the  AFTS  which  they  liked  most  were  (1 ) standardization  of  training;  (2)  replay  capability;  and 
(3)  the  printout  of  performance.  The  AFTS  features  disliked  most  by  students  were  (1)  the  slow  rate  of  the 
voice  transmissions;  (2)  the  lack  of  trend  information  in  sufficient  time  to  use  it;  and  (3)  some  of  the 
emergency  conditions  were  not  realistic.  Comments  made  by  instructors  supported  the  student  list  shown 
above  and  added  one  additional  item.  Tite  simulator/AFTS  communications  system  does  not  permit  the 
instructor  to  give  the  student  directions  from  the  console  without  blanking  out  the  angle  of  attack  auditory 
signals. 

In  summary,  students  and  instructors  liked  the  AFTS  as  a training  device.  Suggested  changes  included 
speeding  up  the  transmission  rate  of  information  given  during  the  GCA  and  improving  message  priorities  to 
be  more  like  real  GCA  controllers.  To  use  the  device  effectively,  students  and  instructors  suggested 
permitting  the  student  to  schedule  his  own  GCA  training  and  use  of  the  device  for  individualized 
instruction.  The  suggested  role  of  the  instructor  was  to  critique  student  performance  and  assist  the  student 
to  overcome  learning  difficulties. 
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Reliability  and  Maintainability  of  the  AFTS 

During  the  initial  formulation  of  the  study,  it  was  anticipated  that  the  system  would  be  subjected  to 
much  use  during  the  operational  evaluation.  Since  the  USAF  was  considering  procurement  of  additional 
systems,  it  seemed  necessary  to  collect  maintenance  information  during  the  conduct  of  the  study.  As  it 
turned  out,  the  conduct  of  the  AFTS  operational  evaluaion  was  very  beneficial  from  the  viewpoint  of 
identification  of  system  hardware  and  software  problems.  The  device  had  been  accepted  by  TAC 
engineering  personnel  prior  to  the  study  and  few  problems  were  expected.  However,  by  conducting  the 
evaluation  in  an  operational  setting,  hardware  and  software  items  that  had  been  successfully  demonstrated 
during  acceptance  testing  were  found  to  have  deficiencies.  The  majority  of  these  items  were  fixed  rapidly 
by  the  contractor  working  in  close  coordination  with  USAF  personnel. 

Appendix  C presents  the  maintenance  problems  identified  during  the  evaluation.  The  major  areas 
included: 

A.  Monitor  Buffer  Interface 

The  monitor  buffer  serves  as  the  interface  device  between  the  AFTS  and  the  GP4B  computer  of  the 
flight  simulator.  During  the  initial  part  of  the  study,  an  intermittent  problem  emerged  which  resulted  in  a 
complete  system  failure  requiring  the  AFTS  to  be  ic-bootcd.  The  frequency  of  system  failures  increased  to 
the  point  where  the  continuation  of  the  evaluation  was  in  question.  The  contractor  was  contacted  and  the 
hardware  problem  was  isolated  and  remedied.  It  appeared  that  certain  insulation  had  deteriorated,  thereby 
resulting  in  intermittent  system  failures. 

B.  Use  of  the  Replay  System 

During  the  first  month  of  the  evaluation,  students  and  instructors  complained  that  the  AFTS  replay 
would  not  work  when  they  wanted  to  use  it.  It  was  found  that  the  problem  was  due  to  a software  error 
which  had  gone  unnoticed  during  earlier  use  of  the  system.  The  software  problem  caused  the  replay  system 
to  be  unusable  when  another  student  was  using  the  AFTS  for  simulator  GCA  training.  This  difficulty  was 
corrected  by  the  contractor  and  USAF  personnel.  No  further  problems  were  experienced. 
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C.  Replay  Display  Screen 

The  4010  Tektronix  displays  which  are  used  for  operator  control  and  replay  were  wearing  out  rapidly 
due  to  use.  This  problem  was  caused  by  the  way  that  the  system  displays  the  same  information  on  the  same 
location  of  the  display  screen.  Thus,  the  display  phosphors  became  worn  and  blank  spots  began  to  appear 
on  the  screen.  Future  systems  should  examine  other  displays  to  minimize  this  problem. 


Recommended  Changes  and  Methods  of  Utilization 


The  recommendations  for  future  changes  and  use  of  the  system  arc  based  to  a large  degree  on  the 
individual  student  training  records.  Throughout  the  training  sorties,  individual  performances  of  adaptively 
trained  students  were  plotted  on  individual  graphs  to  show  progression  in  difficulty  levels.  Students  in  the 
adaptive  group  were  allowed  to  inspect  their  performance  graphs  whenever  they  wished.  Most  students 
monitored  their  charts  on  a daily  basis.  Since  the  control  group  performed  GCAs  at  difficulty  levels 
determined  by  their  trained  counterparts,  only  the  performance  of  the  adaptive  group  was  plotted. 


Figure  6 shows  a plot  of  difficulty  level  achieved  by  number  of  GCA  trials  for  the  evaluation.  Data 
are  shown  for  Student  2,  Student  1 1 , and  the  theoretical  maximum  advancement  rate  that  could  be 
achieved.  Appendix  B contains  graphs  for  all  adaptive  trained  students.  Shown  in  each  graph  are  the  six 
TAC  F4  Syllabus  required  GCAs  discussed  previously.  These  GCAs  were  required  by  all  students  in  both 
groups  and  were  given  during  the  same  sorties  in  training.  Students  in  both  groups  resumed  the 
experimental  part  of  their  GCA  training  on  the  sortie  immediately  following  the  required  training.  The 
mandatory  GCAs  were  as  follows: 


Level  31  - Utility  Hydraulic  Failure  at  eight  miles  from  touchdown,  aircraft  weight  35,000  pounds, 
winds  210  degrees  at  35  knots,  no  turbulence; 

Level  42  - Single  Engine  Failure  (right  engine),  the  engine  was  failed  prior  to  the  start  of  the  GCA, 
aircraft  weight  35,000  pounds,  winds  210  degrees  at  35  knots,  no  turbulence. 

Level  41  - Single  Engine  Failure  (left  engine)  at  eight  miles  from  touchdown,  aircraft  weight  25,000 
pounds,  winds  210  degrees  at  35  knots,  no  turbulence. 

Level  61  - ASR  Approach,  aircraft  weight  35,000  pounds,  winds  calm,  no  turbulence. 

Level  62  - ASR  Approach,  aircraft  weight  35,000  pounds,  winds  calm,  no  turbulence. 


Level  56  - No  Gyro  Approach,  aircraft  weight  35.000  pounds,  winds  210  degrees  at  35  knots,  no 
turbulence. 


Initially,  the  insertion  of  these  required  GCAs  into  the  evaluation  was  viewed  with  some  concern 
since  it  represented  a departure  from  the  planned  experimental  design.  However,  these  GCAs  did  provide 
information  about  the  training  task  which  otherwise  might  not  have  been  collected. 


The  two  students  whose  performance  arc  shown  the  graph  were  selected  as  representative  of  both 
ends  of  the  GCA  performance  continuum  in  the  evaluation.  As  shown  in  Figure  6.  Student  2 did  very  well 
during  training  (in  actuality,  he  was  the  highest  achieving  student  in  the  Adaptive-trained  group).  Student 
11,  who  was  one  of  the  lower  achieving  AFTS  trained  students,  got  off  to  a slow  start  but  in  later  GCA 
trials  was  able  to  establish  a rate  of  advancement  equal  to  that  of  Student  2.  Both  students  had 
advancement  rates  equal  to  the  theoretical  maximum  increment  at  the  end  of  their  training. 


It  is  interesting  to  note  how  each  student  responded  to  the  syllabus  required  GCA  difficulty  levels. 
Examination  of  Figure  6 indicates  that  the  performance  of  the  best  student  was  unaffected  by  the  extreme 
changes  in  difficulty  levels.  Student  1 1 still  was  having  great  difficulty  in  learning  the  initial  GCA  task  (level 
1).  The  introduction  of  the  syllabus  required  difficulty  levels  did  not  hurt  his  performance,  but  neither  did 
it  help.  This  student  presumably  was  trying  to  discover  the  relevant  cues  of  the  GCA  task  when  the  syllabus 
required  GCAs  were  given.  The  additional  cues  presented  during  the  syllabus  required  GCAs  conceivably 
added  to  an  already  unmanageable  number  of  cues.  Thus,  the  student  received  very  little,  if  any,  training 
benefit  from  these  GCAs  because  he  simply  was  not  ready  for  it. 


Recommended  A l-'TS  Changes  Based  on  these  obseivations,  it  may  be  more  advantageous  to  permit 
students  to  practice  the  same  difficulty  level  until  the  GCA  task  is  learned.  The  data  for  Student  1 1 and 
other  low  achievement  students  provide  support  for  this  contention. 


D 

I 

F 

F 

I 

C 

U 

L 

T 

Y 

L 

E 

V 
E 
L 
S 


76 

70 


T 


T 


T 


X 


X 


% 


T theoretical  Maximum  Increment 
2 = Student  Number  Two 
1 1 = Student  Number  Kleven 
S - Syllabus  Required  Training 


0 

» 


10 


20  30  40 


50 


NUMBER  OF  GCA’S 


-M 


$ 

$ 


v>‘ 

* 


Figure  6.  Graph  showing  difficulty  level  achieved  by  selected  adaptive- 
trained  students  during  the  evaluation. 
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Students  should  be  exposed  to  examples  of  different  winds,  aireraft  weights,  and  atmospheric 
turbulence.  However,  the  requirement  for  students  to  adaptively  receive  all  combinations  of  these  variables 
does  not  appear  to  be  necessary  or  efficient  for  training.  Indeed,  data  from  this  study  suggest  that  the 
number  of  difficulty  levels  ha«ed  upon  these  variables  can  be  reduced  considerably  without  affecting  the 
training  value  of  the  AFTS.  Thus,  the  dilficulty  introduced  by  these  variables  appears  to  be  more  imaginary 
titan  real.  The  introduction  of  emergencies  (engine  failures,  flap  failures,  etc.)  or  different  variations  of 
GCAs  (ASR  Approaches  and  No-gyro  Approaches)  appear  to  be  variables  that  make  the  GCA  task  difficult. 
In  these  instances,  the  student  is  forced  to  fly  a GCA  that  is  not  normally  flown.  In  both  cases,  he  must 
transition  such  skills  that  he  has  previously  learned  to  the  new  task. 

In  the  prescr..  valuation,  students  received  32  GCAs.  Vet,  the  best  student  advanced  only  as  far  as 
Difficulty  Level  68.  The  normal  F-4  program  docs  not  have  sufficient  time  allocated  to  GCA  training  to 
permit  students  to  achieve  this  level.  Nonetheless,  it  is  above  Difficulty  Level  30  that  the  student  is  exposed 
to  GCA  emergencies  and  other  variations  of  GCAs.  Thus,  as  the  AFTS  is  designed  at  present,  the  standard 
F-4  student  pilot  will  miss  this  training  unless  the  system  is  manually  set  to  these  levels;  thereby  taking 
away  a potential  advantage  of  adaptive  training. 

In  view  of  these  considerations,  it  is  recommended  that  the  first  thirty  difficulty  levels  he  examined 
with  the  aim  of  reducing  the  number  of  these  steps.  Students  should  he  required  to  master  these  steps 
before  moving  into  the  emergency  GCAs.  The  lower  number  of  difficulty  level  steps  will  permit  students  to 
complete  the  GCA  instruction  program  in  the  AFTS  during  the  normal  F4  training  program.  From  an 
education  standpoint,  this  will  provide  the  student  with  a sense  of  accomplishment  but  it  opens  up  yet 
another  problem  which  must  be  considered.  What  happens  to  the  student  who  reaches  Difficulty  Level  76? 
Does  the  student  stay  at  Level  76  or  should  lie  be  given  other  difficulty  levels?  The  answer  to  this  problem 
has  not  received  adequate  attention  by  other  researchers  and  was  not  considered  in  the  present  AFTS 
design.  Future  adaptive  system  designers  should  considers  additional,  new  approaches  to  this  problem. 

Recommended  Methods  of  Use.  Training  devices  like  the  AFTS  arc  best  suited  for  programs  which 
place  high  emphasis  upon  individual  instruction  ant!  proficiency  advancement.  For  reasons  of  experimental 
control,  the  present  evaluation  did  not  use  the  AFTS  in  this  manner.  However,  the  demonstrated  training 
value  of  the  Logicon  System  on  the  F-4  would  suggest  that  in  the  future  it  should  be  utilized  to  maximize 
training  value. 


IV.  IMPLICATIONS 

The  results  of  the  evaluation  indicated  the  M TS  to  be  an  effective  system  for  training  GCAs.  A 
major  concern  was  to  determine  whether  the  AMS  riovided  any  negative  training.  Throughout  the 
evaluation,  no  information  was  gathered  suggesting  this  to  be  the  case.  Aside  from  the  major  conclusion 
regarding  the  training  effectiveness  of  the  system,  the  data  seemed  to  warrant  implii  ations  in  several  areas. 

Modification  and  Use  of  the  System 

The  evaluation  surfaced  a numbci  of  areas  in  which  modifications  to  the  system  should  be 
considered. 

base  of  Use.  The  present  AITS  should  be  made  easier  to  use  by  nonsophisticated  users.  This 
modification  would  serve  to  -m courage  students  and  instructors  to  use  the  replay  feature  and  other  program 
features  of  the  system.  An  instruction  program  written  in  everyday  language  would  accomplish  the  goal  of 
removing  apprehension  that  new  users  have  about  using  the  system  and  also  would  serve  to  involve  the  user 
in  system  operation. 

Difjiculty  Levels.  The  requirement  for  76  Difficulty  Levels  is  questionable.  The  individual  student 
learning  data  suggest  that  fewer  difficulty  steps  will  be  adequate  for  training.  Indeed,  the  use  of  difficulty 
levels  raises  several  questions  which  should  be  examined. 

How  many  difficulty  levels  are  required  in  the  AFTS?  Although  students  in  the  study  received  more 
GCA  training  than  the  normal  F-4  syllabus  required,  none  of  the  students  reached  the  top  difficulty  level 
(e.g.,  Student  #2  reached  Difficulty  Level  68  in  32  trials).  Thus,  normal  students  probably  will  never  be 


exposed  to  all  ol  the  training  conditions  available  in  system.  Witliout  bypassing  the  adaptive  scheduling 
ieatuic  ol  the  AITS,  lowei  achieving  students  will  tot  he  exposed  to  GCA  emergencies  presently  required 
in  the  1-4  s>  Habits.  Thus,  the  number  of  difficulty  levels  should  he  reduced  to  the  number  of  steps  that  can 
he  accomplished  realistically  in  a training  program.  This  modification  will  result  in  a reduction  in  software 
program  si/e  and  will  set  up  attainable  (lying  ti tuning  goals.  The  exact  number  of  difficulty  levels  will 
depend  upon  the  GCA  training  objectives  identified  by  the  Tactical  Air  Command. 

Which  difficulty  levels  should  he  retained  in  the  AITS?  The  answer  to  this  question  lies  most 
appropriately  in  the  domain  of  instructional  systems  development.  Detailed  specification  of  the  Al-  IS 
training  objectives  will  not  only  reduce  the  number  of  difficulty  levels,  but  will  result  in  identification  of 
the  type  of  GCA  training  to  he  accomplished.  The  data  from  the  evaluation  indicated  that  factors  such  as 
wind  direction  and  speed,  aircraft  weight,  and  atmospheric  tuibulencc  do  not  significantly  effect  pilot 
performance  in  GCAs  after  the  basic  GCA  task  has  been  mastered  in  the  AFTS.  Fmcrgcncy  conditions 
u equire  either  an  aircraft  configuration  change  from  normal  or  which  increase  the  task  workload  of 
t he  pilot  are  factors  that  change  the  real  difficulty  of  the  GCA  task.  Application  of  instructional  systems 
development  principles  will  assist  in  identification  of  the  desired  difficulty  levels.  Additionally,  it  is 
recommended  that  a student  data  bank  system  to  collect  data  on  pilot  performance  for  specific  GCA 
difficulty  levels  will  be  of  significant  benefit  in  determining  which  difficulty  levels  should  be  retained  in  the 
AFTS.  ' 


Adaptive  Scheduling  Algorithm.  The  performance  formula  by  which  the  AFTS  increases  or  decreases 
the  difficulty  level  of  GCA  training  is  called  the  adaptive  algorithm.  Appendix  A shows  the  performance 
algorithm  used  in  the  present  AITS.  However,  it  should  he  understood  that  the  formula  was  based  upon 
analytical  derivation.  As  shown  in  the  formula.  I’ath  Score  and  Gate  Score  receive  equal  weighting  in  the 
scoring  algorithm.  Similarly,  glidcslope,  course,  and  assigned  angle  of  attack  receive  equal  weighting  in  the 
computation  of  Gate  Score  and  I’ath  Score.  Yet.  the  adequacy  of  these  formulas  should  be  empirically 
verified  to  determine  the  contribution  of  each  performance  parameter  to  pnot  performance. 


An  issue  which  is  separate  but  related  to  algorithm  design  concerned  the  number  of  steps  that  pi'ots 
should  be  incremented  or  decremented  based  upon  performance.  The  AITS  presently  will  increase 
difficulty  levels  up  to  a maximum  of  three  levels.  However,  the  efficiency  of  this  limit  is  subject  to 
question.  In  fact,  is  it  necessary  for  the  AFTS  to  set  students  back  in  difficulty  levels  based  upon 
performance?  Several  instructors  in  the  evaluation  suggested  that  the  system  should  increment  but  not 
decrement  students.  The  individual  student  learning  data  shown  in  Appendix  B suggest  that  early  in  GCA 
naming,  pilots  could  be  retained  at  a given  difficulty  level  until  a high  level  of  GCA  peilormar.ee  skill  is 
attained.  Succeeding  changes  in  difficulty  level  then  might  be  increased  (in  relation  to  skill  level)  up  to  a 
maximum  of  four,  five,  or  more  steps.  The  precise  number  of  steps  for  changes  in  difficulty  level  should  be 
reevaluated  so  that  the  AFTS  progression  formula  can  be  made  more  efficient. 


Maintenance  of  1'lving  Skills.  A question  which  is  rarely  considered  by  most  proponents  of  adaptive 
training  or  adaptive  scheduling  is  what  happens  when  the  student  reaches  the  highest  difficulty  level.  The 
AFTS  docs  not  consider  this  question.  Yet  designers  of  automated  systems  in  flying  training  should 
consider  the  issue.  If  the  student  reaches  the  top  step  in  the  program,  how  should  his  skill  be  maintained  in 
a given  task'.’  If  he  stays  at  the  most  difficult  GCA  training  step,  then  skills  on  lower  difficulty  levels  (i.e., 
various  GCA  emergencies)  may  not  be  retain'  d.  Other  approaches  would  suggest  i hat  the  AFTS  program 
should  restart  the  student  at  the  beginning  difficulty  levels  for  emergencies  or  mat  a special  test  program 
incorporating  selected  emergency  GCA  difficulty  levels  be  used.  The  recommended  solution  based  upon 
experience  from  this  evaluation  is  a combination  of  previously  mentioned  approaches.  When  pilots  attain 
the  top  GCA  difficul  y level  of  the  AFTS,  the  software  program  should  automatically  change  the  student  to 
a special  skill  maintenance  program.  This  program  would  consist  of  selected  GCA  emergency  difficulty 
levels.  If  the  student  had  difficulty  with  a particular  emergency,  the  program  would  branch  automatically 
to  the  main  AFTS  training  program  for  remedial  training.  Upon  completion  of  the  remedial  training,  the 
students  GCA  training  would  be  returned  to  the  skill  maintenance  program  for  continuation  training.  Other 
equally  effective  skill  maintenance  programs  can  be  conceived; however,  all  approaches  should  be  carefully 
evaluated  with  respect  to  instructional  objectives. 


Optimizing  System  Use.  As  with  uxist  recent  training  innovations,  automated  training  systems  such  as 
AFTS  are  designed  for  individualized  student  instruction,  lifficicnt  and  cost-effective  use  of  these  systems 
can  be  realized  through  training  programs  that  emphasize  individualized  student  proficiency  advancement. 
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Procurement  of  Additional  Systems 

Should  additional  automated  (light  training  systems  be  procured?  This  question  cannot  be  clearly 
answered  based  upon  data  from  this  evaluation.  The  primary  issue  of  the  study  was  to  determine  training 
effectiveness  and  consequently,  the  experimental  design  reflected  this  consideration.  No  attempt  was  made 
lo  save  time  or  to  effect  other  efficiencies  in  system  use.  Certainly,  the  system  was  effective  for  GCA 
training  in  the  F4  program.  The  AFTS  provides  standardized  training  so  student  skill  capabilities  can  be 
described  quantitatively.  This  intangible  attribute  is  considered  one  of  the  major  values  of  the  system. 
However,  the  issue  of  cost  in  relation  to  application  must  be  considered.  The  AFTS  trains  one  small  but 
specialized  training  area:  GCAs  and  TACAN  approaches.  Vet,  in  the  present  F4  syllabus,  students  receive 
approximately  10-12  GCAs  and  even  fewer  TACAN  approaches  in  the  simulator.  This  fact  is  not  intended 
to  criticize  the  F4  training  program,  but  merely  to  point  out  the  small  amount  of  time  that  the  AFTS 
system  would  bt  used  in  the  norma!  training  program. 

Present  configuration  of  the  system  requires  that  an  operator  or  instructor  be  present  to  monitor  the 
student  during  GCA  training  to  insure  that  the  student  performs  proper  GCA  communications  rather  than 
merely  keying  the  microphone  switch.  Thus,  cost  savings  due  to  elimination  of  personnel  cannot  be 
realized. 

Based  upon  these  considerations,  it  is  recommended  that  the  Tactical  Air  Command  consider  the 
utilization  of  the  device  in  relation  to  its  value  in  standardization  of  training.  The  addition  of  other  training 
areas  such  as  ground  controller  intercepts  (GCI),  ground  attack  radar  (GAR)  navigation,  and  air  refueling 
capabilities  could  provide  increased  utilization  of  the  system.  When  these  training  features  become 
available,  the  improved  AFTS  should  be  evaluated  for  training  value  and  cost-effectiveness. 

Adaptive  Training 

It  should  be  re-emphasized  that  the  present  study  was  not  an  evaluation  of  adaptive  training,  but 
rather  an  operational  system  in  which  adaptive  training  was  only  one  of  its  many  characteristics.  The 
experimental  group,  in  addition  to  the  adaptive  scheduling  based  on  their  own  performance,  received  GCA 
training  with  the  following  characteristics:  (I)  standardized  instruction  for  all  GCAs;  (2)  knowledge  of 
results  from  the  performance  measurement  print-outs;  and  (3)  feedback  using  the  replay  capability. 
Consequently,  it  was  impossible  to  assess  the  contributions  of  each  of  these  characteristics  to  the  training 
effectiveness  of  the  system.  Nevertheless,  there  were  characteristics  of  the  data  which  do  reflect  upon  the 
concept  of  adaptive  training. 

One  of  the  major  requirements  of  adaptive  training  is  that  variations  in  the  adaptive  variable  should 
produce  changes  in  task  difficulty.  It  is  assumed  that  the  resulting  sequence  of  tasks  is  arranged  in  order  of 
increasing  difficulty.  As  indicated  previously,  the  76  levels  of  difficulty  in  the  AFTS  were  defined  with  the 
aid  of  experienced  instructors  from  the  F4  Instructional  System  Development  Team  (ISDT).  That  these 
discrete  steps  actually  represent  a series  of  increasingly  more  difficult  tasks-as  measured  by  actual 
performance-remains  unverified.  The  collection  of  such  data,  using  a sample  of  experienced  pilots,  would 
be  tedious  and  time-consuming.  The  data  available  from  students  within  the  study  arc  confounded  by  the 
fact  that  the  information  was  collected  during  actual  training.  As  indicated  in  die  results,  performance  data 
varied  as  a function  of  training  trials.  It  is  apparent  that  task  difficulty  is  not  varied  sufficiently  in  order  to 
maintain  a constant  level  of  performance.  Such  information  suggests  that  cither:  (a)  the  adaptive  variables 
used  in  this  training  system  do  not  actually  produce  difficulty  changes;  (b  j the  sequence  of  76  tasks  docs 
not  represent  a set  of  increasingly  difficult  tasks;  or  (c)  the  adaptive  scheduling  algorithm  is  inappropriate. 

An  underlying  assumption  of  adaptive  training  is  that  learning  represents  a continuous  process.  On 
each  succesive  trial,  skill  is  incremented  by  a certain  amount.  While  such  continuous  increases  in  skill  level 
may  be  seen  from  group  learning  curves,  it  is  rarely  lire  case  with  individual  learning  curves.  As  stated 
earlier,  the  learning  curves  generated  by  the  12  students  suggest  mastery  of  the  GCA  to  represent  a process 
of  insight.  In  other  words,  students  will  not  advance  until  they  have  mastered  the  concept  of  the  GCA. 
Once  mastered,  however,  students  advance  at  much  the  same  rates.  Such  data  suggest  that  once  the  student 
“learns”  to  fly  the  GCA,  variations  in  wind  velocity,  direction,  aircraft  weight,  and  turbulence  have  little 
effect  on  his  performance.  Only  emergencies  in  which  the  aircraft  configuration  is  dramatically  changed  will 
affect  his  performance. 


It  is  the  opinion  ol'  the  authors  that  while  variations  in  wind,  weight,  and  turbulence  may  add  realism 
to  the  task,  these  changes  based  on  performance  within  the  adaptive  context  do  little  to  facilitate  learning. 
It  is  suggested  that  a random  presentation  of  tiCAs  under  these  conditions  may  be  as  effective  as  the 
present  system  utilizing  adaptive  scheduling  However,  such  a statement  is  a matter  of  conjecture  and  is 
certainly  in  need  of  empirical  validation.  Future  studies  comparing  adaptive  scheduling  with  random  and/or 
fixed  presentation  would  test  the  utility  of  die  adaptive  scheduling  feature. 
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APPENDIX  A:  DESCRIPTION  0I:  APTS  PERFORMANCE  MEASUREMENT, 

ADAPTIVE  SCHEDULING  AND  LEVELS  OF  DIFFICULTY 

Tlie  measures  of  performance  used  in  the  evaluation  will  be  discussed  in  two  categories-those 
delivered  with  the  APTS  and  those  developed  for  the  evaluation. 

AITS  Performance  Measures.  ‘I he  AFTS  provides  three  primary  scores -path  score, gate  score,  and  a 
total  score.  The  path  score  reflects  performance  down  the  glidepath  while  the  gate  score  reflects  aircraft 
parameters  at  decision  height.  The  total  score  is  a combination  of  the  path  and  gate  scores.  The  definitions 
of  each  of  these  measures  is  provided  in  the  System  Operations  Manual  (Logicon,  ll>74).  A summary, 
however,  will  be  presented 

Path  Score.  This  measure  defines  performance  on  the  glidepath  as  a function  of  deviations  from 
glideslope.  centerline,  and  optimum  angle  of  attack.  At  each  iteration  deviations  from  glideslope  and 
centeiline  are  computed  and  categorized  according  to  the  vertical  and  horizontal  zones  shown  in  Figures 
A t and  A-2.  The  basic  measures  are  the  percentages  of  time  within  each  of  the  zones.  Scoring  is  initiated  at 
the  time  of  glideslope  intercept  and  terminates  at  either  decision  height  or  entry  into  any  missed  approach 
zone.  The  path  score  is  computed  accoiding  to  the  following  formula: 

V +11  + a 

Path  Scute  = P = — i ^ + T 


V^  = '-’(OGP)  + ':[  <S\GP|  + "(SBGP)| 

'MOGP)  is  the  percentage  of  time  the  aircraft  was  in  On  Glidepath  Zone. 

'MS  AGP)  is  the  percentage  of  time  the  ait  craft  was  in  Slightly  Above  Gildepath  Zone. 

)SBGP)  is  the  pc  i cent  age  of  time  the  aircraft  was  in  Slightly  Below  Glidepath  Zone. 
lls=  "(11c  < 11+  KHe<M 

"(lie  I ) is  the  pui.tni.ige  of  time  the  aircraft  heading  was  within  1 degree  of  assigned  heading. 

' M !<lk<5)  is  tite  percentage  of  time  the  aircraft  heading  was  greater  than  1 degree  within  5 
degrees  of  assigned  heading 

= "I  IS.  | < AO  A < 'll  3) 

Indicates  the  percentage  ol  time  the  angle  of  attack  was  greater  than  IS.I  units  but  less  than 
20.3  units. 

T = 100  (RAM 

s 

RAI  is  the  Rough  Air  Factor,  a number  between  0 and  !(».  An  Adjusted  Path  Score  (P  ) is 
computed,  as  follows,  when  the  aircraft  fails  to  penetrate  the  Gate  at  the  decision  height. 

P = L(P  ) + 100L 


L = proportion  of  glidepath  completed  prior  to  termination. 

P - oath  score  as  computed  previously. 

(tale  Score.  The  gate  score  represents  a “snapshot”  look  at  performance  at  decision  height.  It  is 
computed  according  to  the  following  fomiula: 

Gate  Score  = G = 1/3(Y  + Z + A - 41  A ) 

s s s s ns 


N = >00-  IY..I 

s I: 


1Y,  I is  the  absolute  lateral  error  tit  the  Gate  ( feet ). 


Preceding  page  blank 
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Z = 100  tz,.i 

s l. 


17  I is  the  absolute  vertical  error  at  the  Gale  (feet). 


A • 100  --  25(  b - 10.2 1) 

b - 10.2  I is  the  absolute  Angle  of  Attack  error  at  the  Gate  (units). 

'i'  = 25  I A I 

s 

is  the  rate  of  change  of  heading  at  the  Gate  (deg/sec). 

A = 25 IA! 

A is  the  rale  of  change  of  angle  of  attack  at  the  Gate  (units/sec). 

Total  Score  The  total  score  is  computed  at  the  end  of  a trial  and  represents  a combination  of  the 
Path  and  Gate  scores.  If  the  cate  is  reached,  the  lollowinu  formula  is  used: 


Total  Score  = I*  + G +100 


where 


I*  = path  score 
G = uate  score 


In  the  event  the  gate  is  not  reached,  the  total  score  is  set  equal  to  the  adjusted  path  score.  P 


Additional  Measures  Several  other  measures  were  derived  from  parameters  computed  within  the 
AITS.  The  path  score  is  probably  the  most  reliable  measure  computed  since  it  utilizes  a large  number  of 
data  points.  Consequently . it  seemed  desirable  to  obtain  measures  on  the  three  major  parameters  defining 
the  path  score  deviations  from  glideslope.  centerline,  and  optimum  angle  of  attack.  Throughout  the 
training  sorties,  scores  were  computed  for  each  of  these  parameters  based  on  the  percentages  of  time  within 
the  different  zones.  Toi  the  criterion  sorties,  the  AITS  software  was  modified  to  provide  more  precise 
root-mean-square  values.  Also,  the  scoring  was  initiated  at  7.5  miles  from  touchdown,  rather  than  at  the 
time  of  glideslope  intersection.  The  formulas  used  are  presented  below. 

(ilidcsli>i>e  Score /RMS  ( ilidcslope  i'.rror.  During  the  training  sorties,  the  glideslope  score  was 
computed  as  follows: 


Glideslope  Score  = '«'  (OGP)  + Gj'TSAGP)  + 'TSBGP)| 


'TOGP)  is  the  percentage  of  time  the  aircraft  was  in  On  Glideslope  Zone. 

(SAGP)  is  the  percentage  of  time  the  aircraft  was  in  Slightly  Above  Glidcpath  Zone. 

'■  (SBGP)  is  the  percentage  of  time  the  aircraft  was  in  Slightly  Below  Glidcpath  Zone. 

The  RMS  Glideslope  h.rror  score  used  during  the  criterion  sorties  was  computed  as  follows: 


V 


N 

- (oi)J 
i = I 


N 


where 


ai  = glideslope  angle  error  in  sample  i 
N = number  of  samples 
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where 


Course  Angle  Score/RMS  Course  Angle  Error.  The  course  angle  score  was  computed  as  follows: 
Course  Angle  Score  = 7t{ OC ) + 5£|9f(SCR)  + '/f<SLC)| 

7c  (OC)  is  the  percentage  of  time  the  aircraft  was  in  On  Course  Zone 
%(SRC)  is  the  percentage  of  time  the  aircraft  was  in  Slightly  Right  of  Course  Zone 
% (SLC)  is  the  percentage  of  time  the  aircraft  was  in  Slightly  Left  of  Course  Zone 
The  RMS  Course  Angle  trror  score  was  computed  as  follows: 


N 

£ (cri)2 

= t 

N 

where 

ai  = Course  angle  error  on  sample  i 
N = Number  of  samples 

Path  Completion  Scon;.  Each  GCA  for  both  the  training  and  criterion  sorties  was  scored  according  to 
whether  or  not  the  path  was  completed;  that  is,  whether  decision  height  was  reached.  A successful 
completion  was  scored  1 , a non-completion,  0. 


Adaptive  Scheduling 

The  AFTS  adaptive  scheduling  program  functioned  in  the  following  manner:  After  the  first  trial  by 
the  student  pilot,  his  total  score  for  the  run  was  used  in  the  adaptive  scheduling  logic  to  determine  the 
appropriate  difficulty  level  for  the  next  trial.  Table  A I shows  the  factors  which  were  used  to  adjust 
difficulty  level  for  each  student 

Table  A I.  Difficulty  Level  Adjustment  Values 


Previous  Run’s 
Step  Number 
Adjustment 
Status 


- (Deere1- tented) 
0 (No  Change) 

+ (Incremented) 


Scoring  Range  (S  represents  Total  Score) 


S<50 


50<S<100 


10(KS<150 


150<S<200 


200<S 


Adjustment  Factors  to  be  added  to,  no  change,  or 
subtracted  from  previous  run’s  Step  number 


-3 

_2 

-1 


_T 

-1 

0 


0 

+1 

+1 


0 
+1 
+ 2 


+ 1 
+2 
+3 


The  difficulty  level  for  each  run  was  based  on  the  total  score  of  the  previous  run.  and  whether  the 
previous  run  had  been  decremented,  unchanged,  or  incremented.  For  example,  suppose  that  a student  was 
just  starting  out  using  the  GCA  adaptive  scheduling  program  and  that  he  received  scores  of  120,  150  and 
200.  Since  the  student  has  not  used  the  system  previously,  then  the  system  has  no  record  of  incrementing 
or  decrementing  the  student.  Therefore,  if  it  is  assumed  that  the  student  started  at  Difficulty  Level  1,  then  a 
score  of  1 20  will  result  in  an  increment  of  + 1 (enter  table  for  0-no  change  and  score  between  100  and  150). 
The  second  trial  will  be  at  Difficulty  Level  2.  For  this  trial,  the  student  received  a score  of  150.  Since  the 
program  logic  now  has  a record  of  the  student  showing  that  he  was  incremented,  the  Difficulty  Level  for 
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the  student  will  be  incremented  by  +2  steps.  The  third  trial  will  be  performed  at  Difficulty  Level  4.  At  the 
completion  of  this  trial,  the  score  of  200  is  examined  for  the  instance  where  the  previous  run  difficulty 
level  has  been  incremented.  In  this  instance,  the  Difficulty  Level  for  the  student  would  be  incremented  +3 
steps.  Thus,  the  fourth  trial  would  be  conducted  at  Difficulty  Level  7.  The  AFTS  program  logic  kept  track 
of  all  students  in  the  program  so  that  the  student  always  started  at  the  Difficulty  Level  where  he  had 
stopped  (even  though  several  days  might  elapse  before  his  next  training  period).  If  the  student  had 
problems  in  training,  then  the  Difficulty  Level  of  lire  trials  would  be  adjusted  downward  to  compensate  for 
his  lack  of  skill. 


GCA  Difficulty  Levels 


Difficulty  was  introduced  into  tiie  GCA  trial  by  automatically  varying  winds  (direction  and  speed), 
aircraft  weight,  atmospheric  turbulence,  and  aircraft  emergencies.  The  initial  layout  of  the  Difficulty  Levels 
was  accomplished  through  the  efforts  of  Logicon  and  the  F-4  Instructional  System  Development  Team 
(ISDT).  The  ordering  of  the  levels  was  based  upon  analytical  examination  and  required  empirical 
verification. 


1.  Wind  Direction  and  Speed.  Wind  direction  relative  to  the  runway  heading  (210)  was  varied  as 
follows: 


Wind  Dircction/Spced 


Wind  Direction/Speed 


210/35  kt 
210/20  kt 
210/10  kt 
Calm 
300/10  kt 


120/10  kt 
300/20  kt 
1 20/20  kt 
030/10  kt 
030/20  kt 


2.  Aircraft  Weight.  Aircraft  weight  and  drag  was  varied  by  changing  the  internal  fuel  load.  The 
following  weights  were  investigated: 


Airciaft  Weight 


35.000 

38.000 

41.000 
43,700 

Reserved  for  future  expansion 


3.  Turbulence.  The  wind  turbulence  factors  affecting  flight  performance  along  the  glidcslope  were: 


Turbulence  Characteristic 


None 

Light 

Moderate 

Heavy 

Reserved  for  future  expansion 


The  heavy  turbulence  that  was  used  was  16  percent  of  the  maximum  amount  of  turbulence  that  was 
available  in  the  simulator. 
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4.  Aircraft  iinciyauics.  The  type  of  malfunction  and  the  point  at  which  the  malfunction  occurred 
from  touchdown  are  shown  in  the  following  list 


O') 

10  32 


None 

Utility  hydiaulie  failure 
flap  failure 
Left  engine  failure 
Right  engine  failure 
Stall  2 aug  failure 
No  g\  ro  approach  procedure,  no 
MlV)i:  failures 
ASR  approach 

Insand  AN/AJB-7  lailure.  No 
gyro  procedure 

Communications  failure  (receiver) 
Reserved  for  future  expansion 


Distance 

from  Touchdown 


10  miles 
10  miles 
K miles 
2 miles 
10  miles 


1 0 miles 


3 miles 


Table  A2  shows  the  76  Difficulty  Levels  that  were  available  in  the  AI  TS.  As  shown  in  the  table,  wind 
was  varied  initially  (difficulty  levels  I 10).  then  weight  and  wind  variables  were  varied  (Difficulty  Levels 
1 1 through  20).  turbulence,  aircraft  weight,  and  wind  were  varied  for  the  next  ten  levels  (Difficulty  Levels 
21  through  30).  Starting  with  Difficulty  Level  31 . aircraft  emergencies  or  degraded  GCA  approaches  were 
introduced  in  combination  with  wind,  aircraft  weight,  and  atmospheric  turbulence  variables. 
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Vertical  glideslope  zones. 


Table  A2.  Mode  1 GCA  Syllabus 


•mymppm 


Step 

Emergency 

Turbulence 

Weight 

Wind 

Difficulty 
Levels  * 
E TWV 

Direction 

Speed 

WIND 

1 

None 

None 

35,  000 

210 

25 

00 

000 

2 

None 

None 

35,  000 

210 

20 

00 

001 

3 

None 

None 

35,  000 

210 

10 

00 

002 

4 

None 

None 

35,  000 

210 

0 

00 

003 

5 

None 

None 

35,  000 

300 

10 

00 

004 

6 

None 

None 

35,  000 

120 

10 

00 

005 

7 

None 

None 

35,  000 

300 

20 

00 

006 

8 

None 

None 

35,  000 

120 

20 

00 

007 

9 

None 

None 

35,  000 

030 

10 

00 

008 

10 

None 

None 

35,  000 

030 

20 

00 

009 

WEIGHT 

1 1 

None 

None 

38,  000 

210 

35 

00 

010 

12 

None 

None 

41 , 000 

210 

35 

00 

020 

13 

None 

None 

43,  700 

210 

35 

00 

030 

14 

None 

None 

43,  700 

210 

0 

00 

033 

15 

None 

None 

43,  700 

300 

10 

00 

034 

16 

None 

None 

38,  000 

120 

10 

00 

015 

17 

None 

None 

41, 000 

120 

20 

00 

027 

18 

None 

None 

33,  000 

030 

10 

00 

018 

19 

None 

None 

41, 000 

030 

20 

00 

029 

20 

None 

None 

43,  700 

030 

20 

00 

039 

TURBULENCE 

2 1 

None 

Light 

35,  000 

210 

35 

00 

100 

22 

None 

Moderate 

35,  000 

210 

35 

00 

200 

23 

None 

Heavy 

35,  000 

210 

35 

00 

300 

24 

None 

Light 

35,  000 

300 

10 

00 

104 

25 

None 

Moderate 

41,  000 

120 

10 

00 

225 

26 

None 

Light 

41, 000 

300 

20 

00 

126 

27 

None 

Moderate 

41,  000 

120 

20 

00 

227 

28 

None 

Moderate 

41, 000 

030 

20 

00 

229 

29 

None 

Heavy 

41,  000 

030 

20 

00 

329 

30 

None 

Heavy 

43,  700 

030 

20 

- 

00 

1 ' 

Note  1.  E represents  emergencies;  T,  turbulence;  W, 

aircraft  weight; 

| 

and  V,  wind  direction  and  speed. 
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Table  A2  ( Continued) 

< 

Wind 

Difficulty 

Levels 

Step 

Emergency 

Turbulence 

Weight 

E TWV 

UTILITY  HY DRAULIC  FAILURES 


None 

Light 

Light 

Moderate 

Heavy 


35. 000 
38,  000 

38. 000 

41. 000 
43,  700 


FLAP  FAILURE 


SINGLE 

41 

03 

None 

42 

04 

None 

43 

03 

Light 

44 

04 

Light 

45 

03 

Light 

•"6 

04 

Light 

47 

03 

Moderate 

48 

04 

Moderate 

49 

03 

Heavy 

50 

04 

Heavy 

38. 000 
38,  000 

38,  000 

41. 000 
41, 000 
43, 700 
43, 700 


STAB  1 AUG  FAILURE 


None 
Light 
Light 
Moder  ale 
Heavy 


35,  000 
38,  000 
38,  000 
41, 000 
43, 700 


MM 

35 

02 

000 

!■ 

10 

02 

114 

HS9 

20 

02 

117 

0 >0 

20 

02 

229 

030 

20 

L.  _ 

02 

339 

K /’«prCB»,ntfl  emergencies;  T,  turbulence;  W,  aircraft  weight; 
and  V,  wind  direction  and  speed. 
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Table  A2  ( Continued) 


I Wind 

Step  Emergency  Turbulence  Weight  I Direction  I Speed 


Difficxilty 
Levels  * 
E TWV 


NO  GYRO  APPROACH 


None 
I iight 
Light 
Moderate 
Heavy 


35,  000 
38, 000 

38. 000 

41. 000 
43, 700 


ASR  APPROACH 


61 

07 

None 

35, 000 

210 

62 

07 

None 

35, 000 

210 

63 

07 

Moderate 

43,  700 

210 

64 

07 

Light 

41, 000 

300 

65 

07 

Moderate 

43,  700 

120 

66 

07 

Light 

41,  000 

300 

67 

07 

Moderate 

43, 700 

120 

68 

07 

Light 

38,  000 

030 

69 

07 

Light 

41,  000 

030 

70 

07 

Moderate 

43, 700 

03  0 

INS  & AN/AJB-7  FAILURE 


None 
Light 
Light 
Mode  rate 
Heavy 


35,  0 
38,  0 
38,  0 
41,  0 


COMMUNICATIONS  FAILURE 


76 

09 

Light 

38,  000 

i 

120 

20 

APPENDIX  B:  TRAINING  AND  CRITERION  SORTIE  DATA 

Table  lil.  Mean  Difficulty  Levels 
for  the  Adaptive  Group  as  a Function 
of  Training  Trials 


Trial 

Mean 

SD 

1 

1 .0000 

0.0000 

2 

1.5000 

0.6455 

3 

1.7500 

0.8292 

4 

2.4167 

1.3819 

5 

3.5833 

2.2158 

6 

5.0833 

3.1480 

7 

6.8333 

3.8261 

8 

7.9167 

4.3677 

9 

9.3333 

5.0056 

10 

1 1 .0000 

6.0000 

11 

13.0000 

6.6081 

12 

15.2500 

7.4288 

13 

1 7.0000 

7.2572 

14 

18.7500 

7.5512 

15 

20.3333 

7.9408 

1 6 

22.5000 

8.1803 

17 

24.5833 

8.4-406 

18 

25.5000 

8.6939 

19 

27.3333 

9.2766 

20 

28.6667 

9.2586 

21 

30.1667 

9.0722 

"IT 

31.8333 

8.8867 

23 

33.3333 

8.9287 

24 

34.6667 

9.0860 

25 

36.0000 

9.0921 

26 

37.2500 

9.1845 

27 

37.8333 

9.6336 

28 

39.1667 

9.8474 

29 

41.0833 

10.6963 

30 

42.4167 

11.6580 

31 

44.7500 

12.1595 

32 

46.8333 

12.6809 

Tabic  B2 


Training 


Mean  Path  Completion  as  a Function 
of  Training  Blocks 

Adaptive  Control 


Block 

X 

SD 

X 

SD 

i 

.208 

.246 

.136 

.308 

-> 

.444 

.343 

.409 

.313 

3 

.541 

.313 

.545 

.294 

4 

.778 

.343 

.788 

.294 

5 

.736 

.250 

.727 

.278 

6 

.695 

.253 

.727 

.278 

7 

.708 

.237 

.621 

.390 

8 

.667 

.297 

.545 

.403 

9 

.500 

.319 

.606 

.269 

10 

.611 

.299 

.697 

.332 

1 1 

.722 

.266 

.758 

.250 

Total 

.601 

.331 

.596 

.362 

See 

Table  1116  for  sample 

sizes. 

Table  B3.  Mean  Glideslope  Score  as  a Function 
of  Training  Blocks 


Adaptive 

Control 

Block 

X 

SD 

X 

SD 

l 

47.40 

12.16 

45.02 

11.50 

2 

52.8S 

12.36 

56.37 

11.36 

3 

57.35 

12.72 

61.54 

8.05 

4 

68.36 

1 1 .69 

62.42 

16.11 

5 

63.45 

8.01 

67.74 

13.91 

6 

55.73 

1 1.18 

67.19 

14.08 

7 

55.63 

16.51 

59.72 

18.68 

8 

57.37 

16.99 

61.52 

13.80 

9 

56.55 

12.27 

56.62 

18.21 

10 

62.43 

13.48 

60.82 

8.14 

11 

57.53 

11.80 

57.28 

7.94 

Total 

57.696 

13.915 

59.659 

14.636 

Table  B4.  Mean  Course  Angle  Score 

as  a Function  of  Training  Blocks 

Adaptive 

Control 

Block 

X 

SD 

X 

SD 

l 

53.56 

20.19 

58.28 

21.07 

2 

60.51 

P.61 

64.31 

16.20 

3 

63.08 

13.13 

65.43 

13.57 

4 

74.20 

11.68 

74.98 

11.05 

5 

68.01 

13.15 

65.32 

11.69 

6 

59.45 

8.86 

69.55 

17.18 

7 

66.17 

14.76 

62.75 

24.26 

8 

75.41 

16.80 

63.85 

17.00 

9 

53.90 

20.83 

68.27 

11.06 

10 

64.05 

18.88 

67.43 

16.53 

11 

72/, 2 

9.49 

69.21 

1 1.89 

Total 

63.706 

16.788 

66.307 

16.643 

Table  B5.  Mean  Angle  of  Attack  Score  i 

as  a 

Function  of  Training  Block 

Adaptive 

Control 

Block 

X 

SD 

X 

SD 

l 

19.75 

12.19 

19.56 

12.75 

2 

22.53 

15.52 

25.01 

13.94 

3 

31.78 

14.40 

28.85 

16.34 

4 

40.31 

18.57 

35.80 

16.58 

5 

32.11 

11.02 

35.61 

16.10 

6 

25.66 

15.19 

30.11 

23.77 

7 

26.68 

12.28 

29.84 

17.91 

8 

33.84 

13.46 

30.70 

13.03 

9 

23.87 

15.67 

26.81 

12.31 

10 

37.25 

16.29 

21.34 

13.99 

11 

34.18 

14.37 

27.33 

13.64 

Total 

29.816 

15.870 

28.269 

16.536 

Tahir  lib.  Mean  Path  Score  as  a Function 
of  Training  Block 


Training 

Block 

Adaptive 

Control 

X 

SD 

X 

SD 

1 

42.52 

1 1.92 

43.01 

10.29 

i 

47.07 

13.37 

51.32 

9.76 

3 

53.74 

10.43 

54.16 

9.88 

4 

65.65 

10.44 

62.38 

12.21 

5 

6 1 .46 

10.53 

63.00 

11.01 

6 

54.14 

8.59 

62.69 

14.46 

7 

59.48 

1 1.53 

61.41 

15.12 

8 

61.25 

1 1 .93 

60.63 

1 1 .30 

9 

54.56 

15.17 

60.57 

10.75 

10 

65.51 

12.85 

59.62 

10.09 

11 

64.69 

9.35 

6 1 .38 

1 1.64 

Total 

57.280 

13.696 

58.203 

13.062 

Tabic  117.  Mean  Gate  Score  as  a Function 
of  Training  Block 


Training 

Block 

Adaptive 

Control 

X 

SD 

X 

SO 

i 

2.30 

16.08 

3.43 

13.06 

2 

18.35 

22.05 

13.33 

16.50 

3 

25.49 

19.14 

24.98 

13.47 

4 

33.77 

27.11 

42.49 

20.99 

5 

27.19 

20.61 

31.65 

12.74 

6 

25.76 

10.81 

25.15 

29.07 

7 

27.02 

18.88 

32.51 

24.86 

8 

28.10 

21.69 

28.76 

22.80 

9 

20.74 

26.47 

27.08 

16.32 

10 

27.71 

18.94 

30.26 

15.07 

11 

35.05 

24.68 

27.68 

17.10 

Total 

24.681 

22.690 

26.120 

21.447 

Tabic  ITS.  Mean  Total  Score  as  a Function 
of  Training  Blocks 


Training 

Block 

Adaptive 

Control 

X 

SD 

X 

SD 

i 

93.57 

43.37 

88.51 

41.06 

2 

139.29 

49.18 

125.64 

45.24 

3 

161.28 

38.42 

154.50 

34.10 

4 

192.47 

44.81 

194.79 

42.06 

5 

171.87 

38.83 

179.34 

43.07 

6 

173.00 

22.99 

170.11 

55.52 

7 

168.17 

39.48 

168.40 

68.50 

8 

171.68 

41.49 

168.70 

50.38 

9 

148.50 

60.96 

168.72 

3 i .73 

10 

181.64 

36.7 1 

168.18 

39.84 

11 

189.77 

39.88 

183.18 

25.81 

Total 

162.800 

50.069 

161.098 

52.998 
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Tilth ' II'1.  Selected  Descriptive  Statistics 
for  Path  Completion  Data  Collected  During 
Criterion  Sorties 


Category • • 

Adaptive 

Control 

X 

SD 

X 

SD 

1st  Some 

.875 

.33  1 

.so  1 

.348 

2nd  So i tie 

.N47 

300 

.758 

.424 

CCA  Controller 

.Sol 

.340 

.8(13 

.348 

AITS  Controller 

sol 

.'4(i 

.758 

.424 

l evel  1 

.474 

143 

.880 

31/ 

Level  3(. 

.300 

.841 

,3oo 

1 evel  4') 

.70S 

.455 

.(>14 

.487 

Total 

.SO  1 

340 

.780 

.414 

• S.,  TahK  IV 1 7 

!•»»  sample  st/ 

*-• 

Table  II in. 

Selected  Descriptive  Statistics 

for  RMS  Glideslope  Error  Data  Collected 

During  Criterion  Sorties 

Adaptive 

Control 

Category 

X 

5D 

X 

SD 

1st  Sortie 

,2oo 

.WT 

.254 

.104 

2nd  Sortie 

.273 

.172 

.272 

.137 

CCA  Controller 

.230 

.124 

.247 

. 1 38 

AI  TS  Controller 

.302 

.144 

.284 

.101 

Level  1 

.210 

.081 

.212 

.077 

Level  30 

.205 

.084 

.211 

.080 

Level  44 

.280 

.225 

.373 

.140 

Total 

,20l> 

. I 04 

.205 

.151 

Table  III  I Selected  Descriptive  Statistics 
for  RMS  Course  Angle  Error  Data  Collected 
During  Criterion  Sorties 


Adaptive 


Category 

X 

SD 

X 

SO 

1st  Sortie 

.400 

• 284 

.377 

.204 

2nd  Sortie 

.345 

.3 1 5 

,3oo 

.238 

GCA  Controller 

.334 

.335 

.244 

.214 

AI  TS  Controller 

.402 

.251 

.444 

.200 

Level  1 

.302 

.214 

.204 

188 

1 evel  30 

.345 

.28'’ 

• 244 

173 

Level  44 

.555 

.33  1 

.55) 

.275 

Total 

.401 

.302 

.371 

.251 
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Tabic  B12.  Selected  Descriptive  Statistics 
for  9 MS  a gle  of  Attack  Error  Data  Collected 
During  Criterion  Sorties 


Adaptive  Control 


Category 

5T 

SD 

X 

SD 

1st  Sortie 

1.676 

.654 

1.787 

.761 

2nd  Sortie 

1.971 

1.078 

1.981 

.763 

GCA  Controller 

1.895 

.968 

1.874 

.786 

AFTS  Controller 

1.752 

.829 

1.895 

.750 

Level  1 

1.653 

.788 

1.820 

.852 

Level  30 

1.758 

.708 

1.918 

.853 

Level  49 

2.060 

1.113 

1.916 

.557 

Total 

1.824 

.904 

1.885 

.768 

Tabic  BI3.  Selected  Descriptive  Statistics  for  Path  Score 
Data  Collected  During  Criterion  Sorties 


Category 

Adaptive 

Control 

X 

SD 

5c 

SD 

1st  Sortie 

76.955 

15.935 

75.982 

15.923 

2nd  Sortie 

75.154 

16.501 

74.889 

14.485 

GCA  Controller 

79.650 

14.467 

80.677 

14.266 

AFTS  Controller 

72.481 

17.113 

70.194 

14.335 

Level  1 

71.740 

14.331 

70.357 

15.466 

Level  30 

86.158 

13.090 

84.543 

13.008 

Level  49 

70.298 

16.162 

71.407 

12.751 

Total 

76.065 

16.249 

75.436 

15.231 

Tabic  B14.  Selected  Descriptive  Statistics  for  Gate  Score 
Data  Collected  During  Criterion  Sorties 

Category 

Adaptive 

Control 

3? 

SD 

X 

SD 

1st  Sortie 

46.992 

28.354 

44.600 

29.733 

2nd  Sortie 

47.958 

28.306 

38.559 

28.918 

GCA  Controller 

48.251 

28.767 

44.974 

31.450 

AFTS  Controller 

46.699 

27.873 

38.185 

26.952 

Level  1 

54.098 

26.550 

31.555 

25.260 

Level  30 

54.575 

24.542 

41.736 

29.891 

Level  49 

33.752 

28.613 

31.448 

29.565 

Total 

47.475 

28.334 

41.580 

29.484 
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Table  HI 5.  Selected  Descriptive  Statistics  for  Total  Score 
Data  Collected  During  Criterion  Sorties 


Adaptive 

Control 

Category 

X 

SO 

X 

SD 

1st  Sortie 

:is.‘)t>o 

40.220 

213.056 

5 1 .406 

2nd  Sortie 

216.706 

40.733 

204.744 

51.013 

tiCA  Cun' roller 

221.414 

51.102 

18.723 

53.757 

AITS  Controller 

214.344 

47.5(>0 

100.077 

48.205 

Level  1 

225.335 

37.107 

217.207 

46.880 

Level  20 

233.7  3ft 

51.770 

210.221 

53.305 

Level  4(> 

104.562 

40.310 

101.623 

50.614 

Total 

217.878 

40.4X0 

200.350 

51.010 

Table  HIT 

Sample  Sizes  for  Training 
Block  Data 

Block 

Adaptive 

Control 

1 

T 

3 

12 

12 

12 

1 1 
1 1 
1 1 

4 

12 

i i 

5 

12 

1 1 

6 

12 

1 1 

7 

12 

i i 

8 

12 

1 1 

0 

12 

1 1 

10 

12 

i i 

1 1 

Total 

12 

132 

1 1 
121 

Table  HI  7 

Sample  Sizes  for  Criterion 
Sortie  Data 

Category 

Adaptive 

Control 

1st  Sortie 

72 

66 

2nd  Sortie 

72 

66 

CCA  Coni  roller 

72 

66 

ALTS  Controller 

72 

M 

level  1 

48 

44 

level  30 

48 

44 

level  40 

48 

44 

Total 

144 

132 

ltfW,W;,i*  ;-»V»«i 
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APPENDIX  C:  MAINTENANCE  PROBLEMS  AND  SYSTEM  DEFICIENCIES 
IDENTIFIED  BY  OPERATOR  AND  MAINTENANCE  PERSONNEL 


Throughout  the  evaluation,  a variety  of  maintenance  problems  and  system  deficiencies  were 
identified  by  the  operator  and  maintenance  personnel.  These  problems  and  the  subsequent  action  taken  are 
identified  as  follows: 


I.  Problem : Voice  takes  too  long  to  give  message.  If  pilot  makes  a drastic  correction,  AFTS  has  to  finish 
message  before  starting  new  message  for  correction. 

Action  Taken : None. 

2 Problem : If  pilot  intercepts  glide  path,  no  scoring  starts. 

Action  Taken.  None. 

3.  Problem : Intermittent  gaps  appear  in  aircraft  trace  and  voice  pauses. 

Action  Taken:  None. 

4 Problem:  4010  screen  wearing  out. 

Action  Taken:  Swapped  main  terminal  with  replay  terminal.  Have  on  hand  part  number  for  new  tube. 

5.  Problem:  Equipment  cabinet  needs  cooling. 

Action  Taken:  Fan  installed  in  AFTS  cabinet  side.  April  74. 

6.  Problem : Equipment  cabinet  needs  hole  for  printer  and  card  reader  power  cords. 

Action  Taken:  Modification  possible.  Suggestion  forwarded  to  contractor. 

7 Problem:  Phrase  “over  landing  threshold”  too  fast. 

Action  Taken:  None. 

8..  Problem:  Monitor  buffer  needs  better  insulation  on  wire. 

Action  Taken:  Forwarded  to  contractor.  At  one  time  they  said  a redesign  was  in  progress. 

9.  Problem:  “Y”  switch  and  interface  wire  needs  to  be  heavier  gage. 

Action  Taken:  Forwarded  to  contractor. 

10.  Problem : All  equipment  in  cabinet  needs  to  be  mounted  better.  (Flimsy). 

Action  Taken:  None.  Contractor  is  aware. 

I I . Problem : Replay  does  not  print  proper  run  number. 

Action  Taken:  None. 

12.  Problem:  Replay  causes  runtime  error  28. 

Action  Taken:  Corrected  - June  74. 

1 3.  Problem : Replay  prints  one  name  on  line  printer. 

Action  Taken:  System  design  at  this  point.  Can  be  changed  at  any  time. 

14.  Problem  : Replay  does  not  always  work. 

Action  Taken:  Corrected  - June  74. 

1 5.  Problem : Need  diagnostics  to  check  AFTS. 

Action  Taken:  None.  TAC  personnel  will  write  our  own. 

16.  Problem:  Some  student  file  run  numbers  are  slipping  through  replay,  or  not  being  deleted. 

Action  Taken:  Corrected  - June 
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17.  froblcm:  Some  student  level  numbers  are  not  being  updated. 

Action  Taken:  Determined  as  operator  error.  Al'TS  program  must  finish  computation  before  it  is  stopped 
or  run  will  not  be  updated.  Suggestion:  wait  for  printer  to  finish  bcofre  “CONTROL  A." 

It  was  also  suggested  that  the  Logicon  ACTS  should  have  a “Malfunction  Clear”  Program  to  be 
initiated  at  touchdown,  to  allow  the  student  to  “Go  Around”  for  subsequent  GCA. 

Logicon  does  not  release  VVSTS  upon  completion  of  a GCA  with  Logicon  inserted  failures  such  as 
engine  flame -out,  utility  hydraulics,  and  Haps  failures. 

After  the  emergency  landing,  the  Logicon  operator  must  “Cntrl  A”  the  machine  to  allow  the 
discrepancies  to  be  cleared  and  then  re  type  into  the  Tektronix  4010  the  pilots  name,  squadron,  date  level, 
etc.,  wl.il"  the  student  pilot  executes  takeoff  and  climbout  again. 

It  was  suggested  that  the  Logicon  system  could  expedite,  and  more  realistically  control  the  GCA 
missions  if  it  would  release  the  malfunction  program  at,  or  just  prior  to.  touchdown  and  later  re-insert  the 
emergencies  required  for  the  succeeding  GCA. 
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